2024-03-26T04:20:36.793 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T04:20:36.793 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018 --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/orig.config.yaml 2024-03-26T04:20:36.824 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:20:36.909 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7622018' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjFUBGFRzeE04nKhP24Q/b3tXFhiqf2FLgvyVoweemiWgsFOoQms9A3hmEg7rWrT2goj041SVDVqqvOZrcvL30= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJAB26fsuA+6G1K/qO2UKQKU4QraxvCjg7fih7yVeO79+yZUDK+RGOjcpzw2CTHwULaS6ZOPCB2yLGygKsewIu8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T04:20:36.909 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T04:20:36.910 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T04:20:36.910 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T04:20:36.912 INFO:teuthology.task.internal:Checking packages... 2024-03-26T04:20:36.936 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T04:20:36.936 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T04:20:36.936 INFO:teuthology.packaging:ref: None 2024-03-26T04:20:36.936 INFO:teuthology.packaging:tag: None 2024-03-26T04:20:36.936 INFO:teuthology.packaging:branch: squid 2024-03-26T04:20:36.936 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:20:36.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T04:20:37.094 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T04:20:37.095 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T04:20:37.101 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T04:20:37.101 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T04:20:37.143 INFO:teuthology.task.internal:Saving configuration 2024-03-26T04:20:37.155 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T04:20:37.177 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:20:37.200 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:15:24.923824', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjFUBGFRzeE04nKhP24Q/b3tXFhiqf2FLgvyVoweemiWgsFOoQms9A3hmEg7rWrT2goj041SVDVqqvOZrcvL30='} 2024-03-26T04:20:37.223 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:15:24.926714', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJAB26fsuA+6G1K/qO2UKQKU4QraxvCjg7fih7yVeO79+yZUDK+RGOjcpzw2CTHwULaS6ZOPCB2yLGygKsewIu8='} 2024-03-26T04:20:37.223 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T04:20:37.229 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T04:20:37.229 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-26T04:20:37.229 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T04:20:37.312 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff2f63fb0d0>, signals=[15]) 2024-03-26T04:20:37.312 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T04:20:37.318 INFO:teuthology.task.internal:Opening connections... 2024-03-26T04:20:37.318 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-03-26T04:20:37.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:20:37.425 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-26T04:20:37.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:20:37.512 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T04:20:37.519 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-03-26T04:20:37.537 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-03-26T04:20:37.537 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-03-26T04:20:37.594 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:20:37.595 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:20:37.595 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:20:37.595 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:20:37.595 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:20:37.595 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:20:37.595 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-03-26T04:20:37.610 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-26T04:20:37.628 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-26T04:20:37.628 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-26T04:20:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-03-26T04:20:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-03-26T04:20:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-03-26T04:20:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-03-26T04:20:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-03-26T04:20:37.685 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T04:20:37.686 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T04:20:37.686 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-26T04:20:37.718 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T04:20:37.729 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T04:20:37.752 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T04:20:37.753 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:20:37.755 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-26T04:20:37.771 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T04:20:37.777 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T04:20:37.777 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:20:37.813 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-26T04:20:37.843 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T04:20:37.894 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T04:20:37.908 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T04:20:37.908 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T04:20:37.908 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:20:37.909 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T04:20:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:20:37.925 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-26T04:20:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:20:37.926 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-03-26T04:20:37.982 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:20:37.982 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-03-26T04:20:37.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-03-26T04:20:37.983 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-361.el9.x86_64 2024-03-26T04:20:37.984 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-361.el9.x86_64 2024-03-26T04:20:37.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-03-26T04:20:39.990 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T04:20:40.082 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T04:20:41.566 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-26T04:20:41.589 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-26T04:20:42.415 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T04:20:42.571 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-26T04:20:44.647 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-26T04:20:44.751 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-26T04:20:49.748 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-26T04:20:50.109 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-26T04:20:51.266 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-26T04:20:52.001 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-26T04:20:52.175 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-26T04:20:57.242 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T04:20:59.457 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:20:59.457 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:20:59.548 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-26T04:20:59.558 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-26T04:20:59.558 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-26T04:20:59.558 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-26T04:20:59.558 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-26T04:20:59.558 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:20:59.558 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-03-26T04:20:59.559 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:20:59.560 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 76 M 2024-03-26T04:20:59.560 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-03-26T04:20:59.560 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-26T04:20:59.927 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:20:59.928 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:20:59.930 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:21:00.080 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:21:00.091 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:21:00.112 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:21:00.270 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 151 kB/s | 66 kB 00:00 2024-03-26T04:21:00.300 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:21:00.333 INFO:teuthology.orchestra.run.smithi129.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T04:21:03.793 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.8 MB/s | 15 MB 00:03 2024-03-26T04:21:07.468 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 3.8 MB/s | 27 MB 00:07 2024-03-26T04:21:08.035 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 4.1 MB/s | 33 MB 00:08 2024-03-26T04:21:08.038 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:21:08.038 INFO:teuthology.orchestra.run.smithi129.stdout:Total 9.0 MB/s | 76 MB 00:08 2024-03-26T04:21:08.706 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-26T04:21:09.075 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-26T04:21:09.075 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-26T04:21:10.071 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-26T04:21:10.072 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-26T04:21:12.378 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-26T04:21:12.544 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-26T04:21:13.176 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:21:13.301 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:21:16.046 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:21:16.177 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:21:17.484 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-26T04:21:19.799 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T04:21:19.800 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T04:21:19.895 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-03-26T04:21:19.901 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-03-26T04:21:19.902 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:21:19.902 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 76 M 2024-03-26T04:21:19.902 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 123 M 2024-03-26T04:21:19.902 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-03-26T04:21:20.527 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 186 kB/s | 66 kB 00:00 2024-03-26T04:21:20.857 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:21:20.969 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:21:25.675 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:21:31.809 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 2.4 MB/s | 27 MB 00:11 2024-03-26T04:21:35.384 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.0 MB/s | 15 MB 00:15 2024-03-26T04:21:52.983 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 1.0 MB/s | 33 MB 00:32 2024-03-26T04:21:52.983 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:21:52.983 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.3 MB/s | 76 MB 00:33 2024-03-26T04:21:53.679 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-03-26T04:21:54.131 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-03-26T04:21:54.131 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-03-26T04:21:54.865 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:21:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:21:54.865 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:21:54.910 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:21:55.161 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-03-26T04:21:55.162 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-03-26T04:21:55.261 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:21:55.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:21:55.262 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:21:55.262 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:21:55.712 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:21:55.713 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-26T04:21:56.072 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:21:56.169 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:21:56.170 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:21:56.170 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:21:56.170 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:21:56.170 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:21:56.170 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T04:21:56.170 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:21:56.170 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:21:56.250 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:21:56.250 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:21:56.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-03-26T04:21:56.289 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-03-26T04:21:56.289 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-03-26T04:21:56.289 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-03-26T04:21:56.289 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-03-26T04:21:56.289 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-03-26T04:21:56.290 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:21:56.290 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-03-26T04:21:56.290 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-03-26T04:21:56.290 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-03-26T04:21:56.290 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:21:56.290 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:21:56.291 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:21:56.291 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:21:56.291 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-03-26T04:21:56.291 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-03-26T04:21:56.292 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:21:56.292 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-03-26T04:21:56.292 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-03-26T04:21:56.292 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:21:56.292 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:21:56.292 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:21:56.293 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:21:56.293 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-03-26T04:21:56.293 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:21:56.293 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:21:56.294 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:21:56.295 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:21:56.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:21:56.939 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-26T04:21:57.780 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-03-26T04:21:58.250 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:21:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-26T04:21:58.282 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-26T04:21:58.297 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.zIz7GoQOpL 2024-03-26T04:21:58.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zIz7GoQOpL 2024-03-26T04:21:58.366 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.zIz7GoQOpL 2024-03-26T04:21:58.406 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:21:58.491 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:21:58.509 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.zIz7GoQOpL is 6KB 2024-03-26T04:21:58.558 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.zIz7GoQOpL 2024-03-26T04:21:58.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:21:58.639 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:21:58.639 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:21:58.639 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:21:58.640 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:21:58.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:21:58.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-26T04:22:01.363 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:22:01.494 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:22:06.569 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:22:06.610 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:22:11.705 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:22:28.803 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T04:22:28.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-26T04:22:28.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:22:42.815 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:22:42.815 INFO:teuthology.orchestra.run.smithi017.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T04:22:42.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:22:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:22:43.246 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:22:43.246 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T04:22:43.246 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T04:22:43.246 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T04:22:43.662 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T04:22:43.662 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:22:43.663 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-03-26T04:22:44.010 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:22:44.111 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:22:44.111 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:22:44.111 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T04:22:44.111 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T04:22:44.112 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T04:22:44.112 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T04:22:44.112 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T04:22:44.112 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T04:22:44.194 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T04:22:44.195 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T04:22:44.195 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-03-26T04:22:44.232 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 68.el9 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8264254 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T04:22:44.233 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:22:44.234 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T04:22:44.235 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T04:22:44.235 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-432.el9.x86_64 2024-03-26T04:22:44.235 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T04:22:44.943 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-03-26T04:22:46.419 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T04:22:46.456 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-03-26T04:22:46.458 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-03-26T04:22:46.474 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.Z9epa0hxeN 2024-03-26T04:22:46.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z9epa0hxeN 2024-03-26T04:22:46.555 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.Z9epa0hxeN 2024-03-26T04:22:46.699 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.Z9epa0hxeN is 6KB 2024-03-26T04:22:46.750 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.Z9epa0hxeN 2024-03-26T04:22:46.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T04:22:46.833 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T04:22:46.833 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T04:22:46.833 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T04:22:46.833 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T04:22:46.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T04:22:46.999 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-03-26T04:23:03.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-26T04:23:12.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-26T04:23:12.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:23:17.006 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T04:23:17.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-26T04:23:17.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:23:30.850 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-03-26T04:23:35.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-03-26T04:23:42.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-26T04:23:42.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:23:44.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-26T04:23:44.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:23:45.162 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-26T04:23:45.636 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-26T04:23:45.636 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:23:45.637 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-26T04:23:45.651 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:23:45.651 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:23:45.651 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:23:45.651 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:23:46.652 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T04:23:46.652 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:23:46.759 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-26T04:23:46.794 DEBUG:teuthology.parallel:result is None 2024-03-26T04:24:44.460 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T04:24:56.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-26T04:24:56.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:24:56.773 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-03-26T04:24:57.196 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-26T04:24:57.196 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T04:24:57.197 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-03-26T04:24:57.212 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-432.el9.x86_64 2024-03-26T04:24:57.213 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T04:24:57.213 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T04:24:57.213 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T04:24:58.214 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T04:24:58.214 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T04:24:58.297 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-03-26T04:24:58.326 DEBUG:teuthology.parallel:result is None 2024-03-26T04:24:58.326 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T04:24:58.335 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T04:24:58.335 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:24:58.338 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T04:24:58.358 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T04:24:58.392 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T04:24:58.452 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T04:24:58.453 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:24:58.455 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T04:24:58.486 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T04:24:58.497 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T04:24:58.497 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:24:58.519 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T04:24:58.552 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:24:58.563 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:24:58.566 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:24:58.577 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T04:24:58.579 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T04:24:58.590 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T04:24:58.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:24:58.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T04:24:58.640 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T04:24:58.657 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T04:24:58.657 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:24:58.678 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T04:24:58.696 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:24:58.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:24:58.826 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:24:58.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:24:58.952 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:24:58.952 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:24:59.020 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:24:59.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T04:24:59.120 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:24:59.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T04:24:59.248 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:24:59.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T04:24:59.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-03-26T04:24:59.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-26T04:24:59.366 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:24:59.403 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:24:59.746 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T04:24:59.753 INFO:teuthology.task.internal:Starting timer... 2024-03-26T04:24:59.754 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T04:24:59.792 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T04:24:59.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-03-26T04:24:59.891 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-03-26T04:24:59.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-03-26T04:24:59.957 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-03-26T04:24:59.959 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T04:24:59.959 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-03-26T04:24:59.984 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-03-26T04:24:59.985 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-03-26T04:25:00.010 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-03-26T04:25:00.010 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-03-26T04:25:00.010 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T04:25:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:25:00.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T04:25:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:25:00.081 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T04:25:00.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:25:00.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-03-26T04:25:00.178 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T04:25:00.188 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-26T04:25:00.193 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-26T04:25:00.310 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T04:25:00.326 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T04:25:00.327 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T04:29:15.339 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-03-26T04:29:15.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-26T04:29:15.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:15.428 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-03-26T04:29:15.504 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-03-26T04:29:15.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-26T04:29:15.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:29:15.590 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-26T04:29:15.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-26T04:29:15.674 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T04:29:15.684 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T04:29:15.684 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:29:15.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:29:15.687 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:29:15.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:29:15.716 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:29:15.732 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:29:15.746 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:29:15.762 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-03-26T04:29:15.762 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:29:15.776 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-03-26T04:29:15.791 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-03-26T04:29:15.792 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:29:15.805 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-03-26T04:29:15.808 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:29:15.821 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:29:15.837 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:29:15.861 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:29:15.864 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:29:15.865 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-03-26T04:29:15.865 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.865 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.865 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.865 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.889 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:29:15.892 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:29:15.892 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-26T04:29:15.892 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.892 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.892 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.892 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:29:15.893 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T04:29:15.987 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T04:29:15.987 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-26T04:29:15.987 INFO:tasks.cephadm:Cluster fsid is 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:29:15.988 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T04:29:15.988 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.17', 'mon.c': '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]', 'mon.b': '172.21.15.129'} 2024-03-26T04:29:15.988 INFO:tasks.cephadm:First mon is mon.a on smithi017 2024-03-26T04:29:15.988 INFO:tasks.cephadm:First mgr is y 2024-03-26T04:29:15.988 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T04:29:15.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo hostname $(hostname -s) 2024-03-26T04:29:16.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-03-26T04:29:16.043 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-26T04:29:16.043 DEBUG:teuthology.orchestra.run.smithi017:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:29:16.286 INFO:teuthology.orchestra.run.smithi017.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 26 04:29 /home/ubuntu/cephtest/cephadm 2024-03-26T04:29:16.286 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T04:29:16.602 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 26 04:29 /home/ubuntu/cephtest/cephadm 2024-03-26T04:29:16.602 DEBUG:teuthology.orchestra.run.smithi017:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:29:16.618 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T04:29:16.643 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-26T04:29:16.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-26T04:29:16.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-26T04:29:17.482 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T04:29:17.516 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout: "repo_digests": [ 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-03-26T04:29:39.712 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:29:39.812 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-26T04:29:39.812 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-26T04:29:39.812 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-26T04:29:39.812 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-03-26T04:29:39.812 INFO:teuthology.orchestra.run.smithi129.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-26T04:29:39.812 INFO:teuthology.orchestra.run.smithi129.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-26T04:29:39.813 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-03-26T04:29:39.813 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-26T04:29:39.830 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2024-03-26T04:29:39.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-03-26T04:29:39.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /etc/ceph 2024-03-26T04:29:39.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-03-26T04:29:39.955 INFO:tasks.cephadm:Writing seed config... 2024-03-26T04:29:39.956 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-26T04:29:39.956 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T04:29:39.956 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T04:29:39.956 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-26T04:29:39.957 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T04:29:39.957 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T04:29:39.957 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T04:29:39.957 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T04:29:39.957 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T04:29:39.957 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:29:39.957 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T04:29:39.981 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 67a15696-eb29-11ee-95d0-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T04:29:39.982 DEBUG:teuthology.orchestra.run.smithi017:mon.a> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T04:29:40.024 DEBUG:teuthology.orchestra.run.smithi017:mgr.y> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T04:29:40.067 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T04:29:40.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 67a15696-eb29-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.17 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:29:40.267 INFO:teuthology.orchestra.run.smithi017.stderr:-------------------------------------------------------------------------------- 2024-03-26T04:29:40.267 INFO:teuthology.orchestra.run.smithi017.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '67a15696-eb29-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.17', '--skip-admin-label'] 2024-03-26T04:29:40.289 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.4-dev 2024-03-26T04:29:40.291 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying podman|docker is present... 2024-03-26T04:29:40.312 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.4-dev 2024-03-26T04:29:40.313 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying lvm2 is present... 2024-03-26T04:29:40.314 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying time synchronization is in place... 2024-03-26T04:29:40.320 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:29:40.326 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-03-26T04:29:40.333 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: enabled 2024-03-26T04:29:40.339 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: active 2024-03-26T04:29:40.339 INFO:teuthology.orchestra.run.smithi017.stderr:Unit chronyd.service is enabled and running 2024-03-26T04:29:40.340 INFO:teuthology.orchestra.run.smithi017.stderr:Repeating the final host check... 2024-03-26T04:29:40.360 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.4-dev 2024-03-26T04:29:40.362 INFO:teuthology.orchestra.run.smithi017.stderr:podman (/bin/podman) version 4.9.4 is present 2024-03-26T04:29:40.362 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl is present 2024-03-26T04:29:40.362 INFO:teuthology.orchestra.run.smithi017.stderr:lvcreate is present 2024-03-26T04:29:40.368 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:29:40.374 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-03-26T04:29:40.380 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: enabled 2024-03-26T04:29:40.386 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: active 2024-03-26T04:29:40.387 INFO:teuthology.orchestra.run.smithi017.stderr:Unit chronyd.service is enabled and running 2024-03-26T04:29:40.387 INFO:teuthology.orchestra.run.smithi017.stderr:Host looks OK 2024-03-26T04:29:40.387 INFO:teuthology.orchestra.run.smithi017.stderr:Cluster fsid: 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:29:40.387 INFO:teuthology.orchestra.run.smithi017.stderr:Acquiring lock 139695246659488 on /run/cephadm/67a15696-eb29-11ee-95d0-87774f69a715.lock 2024-03-26T04:29:40.387 INFO:teuthology.orchestra.run.smithi017.stderr:Lock 139695246659488 acquired on /run/cephadm/67a15696-eb29-11ee-95d0-87774f69a715.lock 2024-03-26T04:29:40.387 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying IP 172.21.15.17 port 3300 ... 2024-03-26T04:29:40.388 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying IP 172.21.15.17 port 6789 ... 2024-03-26T04:29:40.388 INFO:teuthology.orchestra.run.smithi017.stderr:Base mon IP is 172.21.15.17, final addrv is [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-03-26T04:29:40.390 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.17 metric 100 2024-03-26T04:29:40.390 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.17 metric 100 2024-03-26T04:29:40.392 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T04:29:40.392 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T04:29:40.393 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T04:29:40.395 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T04:29:40.395 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-26T04:29:40.395 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-26T04:29:40.395 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T04:29:40.395 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:d992/64 scope link noprefixroute 2024-03-26T04:29:40.395 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-26T04:29:40.396 INFO:teuthology.orchestra.run.smithi017.stderr:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2024-03-26T04:29:40.396 INFO:teuthology.orchestra.run.smithi017.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T04:29:40.397 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T04:29:40.500 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Getting image source signatures 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-26T04:29:40.929 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-26T04:29:41.057 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Writing manifest to image destination 2024-03-26T04:29:41.168 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-26T04:29:41.552 INFO:teuthology.orchestra.run.smithi017.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-26T04:29:42.125 INFO:teuthology.orchestra.run.smithi017.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-26T04:29:42.125 INFO:teuthology.orchestra.run.smithi017.stderr:Extracting ceph user uid/gid from container image... 2024-03-26T04:29:42.559 INFO:teuthology.orchestra.run.smithi017.stderr:stat: 167 167 2024-03-26T04:29:43.014 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial keys... 2024-03-26T04:29:43.389 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQA3TwJm2Mk0EhAAjaiXKI0EA0kne/ROSs2nsw== 2024-03-26T04:29:52.136 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQA/TwJmla+SEBAADlK37IQWFyif4YX5iN0wmA== 2024-03-26T04:29:55.810 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQBDTwJm9IO/KhAAxNTJe+TMhA7vwsO19oQFcg== 2024-03-26T04:29:56.400 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial monmap... 2024-03-26T04:29:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:29:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-26T04:29:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:29:56.758 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:29:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:monmaptool for a [v2:172.21.15.17:3300,v1:172.21.15.17:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:29:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:setting min_mon_release = octopus 2024-03-26T04:29:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: set fsid to 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:29:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:29:57.239 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:29:57.239 INFO:teuthology.orchestra.run.smithi017.stderr:Creating mon... 2024-03-26T04:29:58.087 INFO:teuthology.orchestra.run.smithi017.stderr:create mon.a on 2024-03-26T04:29:58.293 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T04:29:58.445 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-67a15696-eb29-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-67a15696-eb29-11ee-95d0-87774f69a715.target. 2024-03-26T04:29:58.445 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-67a15696-eb29-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-67a15696-eb29-11ee-95d0-87774f69a715.target. 2024-03-26T04:29:58.735 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to reset failed state of unit ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T04:29:58.742 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph-67a15696-eb29-11ee-95d0-87774f69a715.target.wants/ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-67a15696-eb29-11ee-95d0-87774f69a715@.service. 2024-03-26T04:29:59.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:29:59 smithi017 podman[24440]: 2024-03-26 04:29:59.348786517 +0000 UTC m=+0.325392374 container start 01e8dba0ccd8e69473c93e94476162e0e9ea32195af3ad55eb2f94b22d0f1cee (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., release=754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD) 2024-03-26T04:29:59.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:29:59 smithi017 ceph-mon[24459]: mkfs 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:29:59.475 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-03-26T04:29:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-03-26T04:29:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-03-26T04:29:59.481 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to enable service . firewalld.service is not available 2024-03-26T04:29:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mon to start... 2024-03-26T04:29:59.482 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mon... 2024-03-26T04:29:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:29:59 smithi017 ceph-mon[24459]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:29:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:29:59 smithi017 bash[24440]: 01e8dba0ccd8e69473c93e94476162e0e9ea32195af3ad55eb2f94b22d0f1cee 2024-03-26T04:29:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:29:59 smithi017 systemd[1]: Started Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: cluster: 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: id: 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: services: 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.621843s) 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: data: 2024-03-26T04:30:00.020 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-26T04:30:00.021 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-26T04:30:00.021 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-26T04:30:00.021 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: pgs: 2024-03-26T04:30:00.021 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:00.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:00 smithi017 ceph-mon[24459]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:30:00.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:00 smithi017 ceph-mon[24459]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-03-26T04:30:00.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:00 smithi017 ceph-mon[24459]: fsmap 2024-03-26T04:30:00.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:00 smithi017 ceph-mon[24459]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:30:00.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:00 smithi017 ceph-mon[24459]: mgrmap e1: no daemons active 2024-03-26T04:30:00.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:00 smithi017 ceph-mon[24459]: from='client.? 172.21.15.17:0/3031461814' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T04:30:01.548 INFO:teuthology.orchestra.run.smithi017.stderr:mon is available 2024-03-26T04:30:01.548 INFO:teuthology.orchestra.run.smithi017.stderr:Assimilating anything we can from ceph.conf... 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [global] 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: fsid = 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [mgr] 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-26T04:30:02.072 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-26T04:30:02.073 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:02.073 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [osd] 2024-03-26T04:30:02.073 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-26T04:30:02.073 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-26T04:30:02.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:02 smithi017 ceph-mon[24459]: from='client.? 172.21.15.17:0/4139983125' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:30:02.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:02 smithi017 ceph-mon[24459]: from='client.? 172.21.15.17:0/4139983125' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:30:02.620 INFO:teuthology.orchestra.run.smithi017.stderr:Generating new minimal ceph.conf... 2024-03-26T04:30:03.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:03 smithi017 ceph-mon[24459]: from='client.? 172.21.15.17:0/275567945' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:30:03.679 INFO:teuthology.orchestra.run.smithi017.stderr:Restarting the monitor... 2024-03-26T04:30:03.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:03 smithi017 systemd[1]: Stopping Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:30:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:03 smithi017 bash[24690]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon.a" found: no such container 2024-03-26T04:30:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a[24455]: 2024-03-26T04:30:03.828+0000 7f7017989700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:30:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a[24455]: 2024-03-26T04:30:03.828+0000 7f7017989700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T04:30:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:03 smithi017 podman[24701]: 2024-03-26 04:30:03.850310874 +0000 UTC m=+0.120367636 container died 01e8dba0ccd8e69473c93e94476162e0e9ea32195af3ad55eb2f94b22d0f1cee (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, release=754, GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-26T04:30:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 podman[24701]: 2024-03-26 04:30:04.229454989 +0000 UTC m=+0.499511750 container cleanup 01e8dba0ccd8e69473c93e94476162e0e9ea32195af3ad55eb2f94b22d0f1cee (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, ceph=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., name=centos-stream, version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, distribution-scope=public, release=754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container) 2024-03-26T04:30:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 bash[24701]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:30:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 bash[24728]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon.a" found: no such container 2024-03-26T04:30:04.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 podman[24713]: 2024-03-26 04:30:04.522092379 +0000 UTC m=+0.672253635 container remove 01e8dba0ccd8e69473c93e94476162e0e9ea32195af3ad55eb2f94b22d0f1cee (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.openshift.tags=base centos centos-stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, architecture=x86_64) 2024-03-26T04:30:04.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:30:04.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 systemd[1]: Stopped Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:30:04.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 systemd[1]: Starting Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:30:04.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 podman[24795]: 2024-03-26 04:30:04.747780718 +0000 UTC m=+0.107472764 container create e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-03-26T04:30:04.838 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 podman[24795]: 2024-03-26 04:30:04.657605414 +0000 UTC m=+0.017297474 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-26T04:30:05.011 INFO:teuthology.orchestra.run.smithi017.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-26T04:30:05.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 podman[24795]: 2024-03-26 04:30:04.899900136 +0000 UTC m=+0.259592184 container init e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, release=754, name=centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vcs-type=git) 2024-03-26T04:30:05.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 podman[24795]: 2024-03-26 04:30:04.902665454 +0000 UTC m=+0.262357500 container start e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, RELEASE=HEAD, vcs-type=git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, architecture=x86_64) 2024-03-26T04:30:05.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:30:05.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-26T04:30:05.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: pidfile_write: ignore empty --pid-file 2024-03-26T04:30:05.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: load: jerasure load: lrc 2024-03-26T04:30:05.137 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: RocksDB version: 6.15.5 2024-03-26T04:30:05.137 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-26T04:30:05.137 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Compile date Apr 18 2022 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: DB SUMMARY 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: DB Session ID: 0IF3MXZ0ER5CIWSJCUVN 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82633 ; 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.env: 0x5635266c6860 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.fs: Posix File System 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.info_log: 0x563528c7bdc0 2024-03-26T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.statistics: (nil) 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.use_fsync: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.db_log_dir: 2024-03-26T04:30:05.139 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.write_buffer_manager: 0x563528d6c2a0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:30:05.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.unordered_write: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.row_cache: None 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.wal_filter: None 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.preserve_deletes: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:30:05.141 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:30:05.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_open_files: -1 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Compression algorithms supported: 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kZSTD supported: 0 2024-03-26T04:30:05.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.merge_operator: 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_filter: None 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:30:05.144 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563528c47d00) 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: hash_index_allow_collision: 1 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 1 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x563528cb2170 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:30:05.145 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-03-26T04:30:05.146 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 4 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression: NoCompression 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.num_levels: 7 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:30:05.147 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:30:05.148 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:30:05.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.arena_block_size: 4194304 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-26T04:30:05.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:30:05.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.ttl: 2592000 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:30:05.152 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427404928053, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427404928733, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79183, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77446, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71755, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711427404, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "927481eb-d35f-4893-a7bb-b84761848673", "db_session_id": "0IF3MXZ0ER5CIWSJCUVN"}} 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711427404929078, "job": 1, "event": "recovery_finished"} 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563528c98a80 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: DB pointer 0x563528ca8000 2024-03-26T04:30:05.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.154 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 26.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 26.20 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:30:05.155 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 166.3 0.00 0.00 1 0.000 0 0 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:30:05.156 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 25.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???) e1 preinit fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).mds e1 new map 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T04:30:05.157 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: fsmap 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:04 smithi017 ceph-mon[24814]: mgrmap e1: no daemons active 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:05 smithi017 bash[24795]: e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 2024-03-26T04:30:05.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:05 smithi017 systemd[1]: Started Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:30:06.135 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-26T04:30:06.135 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:30:06.136 INFO:teuthology.orchestra.run.smithi017.stderr:Creating mgr... 2024-03-26T04:30:06.136 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying port 9283 ... 2024-03-26T04:30:06.310 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to reset failed state of unit ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T04:30:06.317 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph-67a15696-eb29-11ee-95d0-87774f69a715.target.wants/ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-67a15696-eb29-11ee-95d0-87774f69a715@.service. 2024-03-26T04:30:06.614 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:06 smithi017 systemd[1]: Starting Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:30:06.901 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:06 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3046483580' entity='client.admin' 2024-03-26T04:30:06.901 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:06 smithi017 podman[25028]: 2024-03-26 04:30:06.609336105 +0000 UTC m=+0.018230912 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-26T04:30:06.901 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:06 smithi017 podman[25028]: 2024-03-26 04:30:06.800153654 +0000 UTC m=+0.209048480 container create 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-type=git) 2024-03-26T04:30:07.123 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-03-26T04:30:07.130 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-03-26T04:30:07.130 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-03-26T04:30:07.130 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to enable service . firewalld.service is not available 2024-03-26T04:30:07.138 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-03-26T04:30:07.145 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-03-26T04:30:07.145 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-03-26T04:30:07.146 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-26T04:30:07.146 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr to start... 2024-03-26T04:30:07.146 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr... 2024-03-26T04:30:07.271 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:07 smithi017 podman[25028]: 2024-03-26 04:30:07.01110652 +0000 UTC m=+0.420001308 container init 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, ceph=True) 2024-03-26T04:30:07.271 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:07 smithi017 podman[25028]: 2024-03-26 04:30:07.013706016 +0000 UTC m=+0.422600795 container start 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, ceph=True, io.buildah.version=1.19.8, version=8, build-date=2022-05-03T08:36:31.336870) 2024-03-26T04:30:07.272 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:07 smithi017 bash[25028]: 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec 2024-03-26T04:30:07.272 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:07 smithi017 systemd[1]: Started Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:30:07.522 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:07.515+0000 7fe450cae000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "67a15696-eb29-11ee-95d0-87774f69a715", 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T04:30:07.650 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T04:30:07.651 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T04:30:07.652 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-03-26T04:30:07.653 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-03-26T04:29:59.397116+0000", 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T04:30:07.654 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:07.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:07 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3457720042' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:30:07.942 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:07.609+0000 7fe450cae000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:30:08.245 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (1/15)... 2024-03-26T04:30:08.555 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.229+0000 7fe450cae000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:30:08.555 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.320+0000 7fe450cae000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:30:08.555 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.404+0000 7fe450cae000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:30:08.831 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.554+0000 7fe450cae000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:30:08.831 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.636+0000 7fe450cae000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:30:08.831 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.746+0000 7fe450cae000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:30:08.831 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.830+0000 7fe450cae000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:30:09.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:08.994+0000 7fe450cae000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:30:09.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:09 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:09.080+0000 7fe450cae000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:30:10.393 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:10 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:10.138+0000 7fe450cae000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:30:10.393 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:10 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:10.272+0000 7fe450cae000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:30:10.663 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:10 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:10.471+0000 7fe450cae000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:30:10.663 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:10 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:10.560+0000 7fe450cae000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:30:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:10.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "67a15696-eb29-11ee-95d0-87774f69a715", 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:10.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T04:30:10.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T04:30:10.848 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:10.849 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:10.850 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-03-26T04:29:59.397116+0000", 2024-03-26T04:30:10.851 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:10.851 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:10.851 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T04:30:10.851 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:10.934 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:10 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:10.700+0000 7fe450cae000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:30:11.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:10 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3012793108' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:30:11.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:11 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:11.025+0000 7fe450cae000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:30:11.445 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (2/15)... 2024-03-26T04:30:11.510 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:11 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:11.250+0000 7fe450cae000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:30:11.510 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:11 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:11.335+0000 7fe450cae000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:30:11.510 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:11 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:11.509+0000 7fe450cae000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:30:12.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:12 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:12.441+0000 7fe450cae000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:30:13.051 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:12 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:12.740+0000 7fe450cae000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:30:13.051 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:12 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:12.834+0000 7fe450cae000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:30:13.051 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:12 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:12.986+0000 7fe450cae000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:30:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: Activating manager daemon y 2024-03-26T04:30:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: mgrmap e2: y(active, starting, since 0.00277608s) 2024-03-26T04:30:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:30:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: Manager daemon y is now available 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' 2024-03-26T04:30:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:13 smithi017 ceph-mon[24814]: from='mgr.14100 172.21.15.17:0/1859410928' entity='mgr.y' 2024-03-26T04:30:13.997 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:13.997 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:13.997 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "67a15696-eb29-11ee-95d0-87774f69a715", 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-03-26T04:30:13.998 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T04:30:13.999 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T04:30:14.000 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-03-26T04:29:59.397116+0000", 2024-03-26T04:30:14.001 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:14.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:14.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T04:30:14.002 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:14 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4047861696' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:30:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:14 smithi017 ceph-mon[24814]: mgrmap e3: y(active, since 1.0072s) 2024-03-26T04:30:14.476 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (3/15)... 2024-03-26T04:30:17.164 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:16 smithi017 ceph-mon[24814]: mgrmap e4: y(active, since 2s) 2024-03-26T04:30:17.166 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "67a15696-eb29-11ee-95d0-87774f69a715", 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-03-26T04:30:17.167 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T04:30:17.168 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T04:30:17.172 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-03-26T04:30:17.173 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-03-26T04:29:59.397116+0000", 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T04:30:17.174 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:17.699 INFO:teuthology.orchestra.run.smithi017.stderr:mgr is available 2024-03-26T04:30:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:17 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4235886293' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [global] 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: fsid = 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [mgr] 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-26T04:30:18.273 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-03-26T04:30:18.274 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [osd] 2024-03-26T04:30:18.274 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-26T04:30:18.274 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-26T04:30:18.839 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling cephadm module... 2024-03-26T04:30:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:18 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3232778162' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:30:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:18 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3232778162' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:30:20.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:19 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3845129608' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T04:30:20.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:19 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setuser ceph since I am not root 2024-03-26T04:30:20.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:19 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setgroup ceph since I am not root 2024-03-26T04:30:20.533 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:20 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:20.442+0000 7f11ec9a3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:30:20.533 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:20 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:20.533+0000 7f11ec9a3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:30:21.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:20 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3845129608' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T04:30:21.151 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:20 smithi017 ceph-mon[24814]: mgrmap e5: y(active, since 6s) 2024-03-26T04:30:21.223 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:21.223 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-26T04:30:21.223 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-03-26T04:30:21.223 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-26T04:30:21.223 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-26T04:30:21.223 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:21.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.150+0000 7f11ec9a3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:30:21.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.244+0000 7f11ec9a3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:30:21.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.329+0000 7f11ec9a3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:30:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for the mgr to restart... 2024-03-26T04:30:21.711 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr epoch 5... 2024-03-26T04:30:21.764 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.480+0000 7f11ec9a3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:30:21.764 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.562+0000 7f11ec9a3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:30:21.764 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.673+0000 7f11ec9a3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:30:21.764 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.763+0000 7f11ec9a3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:30:22.016 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3410495265' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:30:22.016 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:21.927+0000 7f11ec9a3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:30:22.017 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:22 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:22.016+0000 7f11ec9a3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:30:23.388 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:23.063+0000 7f11ec9a3000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:30:23.388 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:23.194+0000 7f11ec9a3000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:30:23.388 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:23.387+0000 7f11ec9a3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:30:23.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:23.476+0000 7f11ec9a3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:30:23.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:23.615+0000 7f11ec9a3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:30:24.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:23.935+0000 7f11ec9a3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:30:24.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:24 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:24.158+0000 7f11ec9a3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:30:24.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:24 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:24.244+0000 7f11ec9a3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:30:24.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:24 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:24.417+0000 7f11ec9a3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:30:25.648 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:25 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:25.348+0000 7f11ec9a3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:30:25.649 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:25 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:25.648+0000 7f11ec9a3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:30:25.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:25 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:25.743+0000 7f11ec9a3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:30:25.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:25 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:25.895+0000 7f11ec9a3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:30:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-26T04:30:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "initialized": true 2024-03-26T04:30:26.947 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: Active manager daemon y restarted 2024-03-26T04:30:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: Activating manager daemon y 2024-03-26T04:30:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T04:30:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: mgrmap e6: y(active, starting, since 0.0297227s) 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: Manager daemon y is now available 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:27.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:27.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:30:27.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:26 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:30:27.500 INFO:teuthology.orchestra.run.smithi017.stderr:mgr epoch 5 is available 2024-03-26T04:30:27.500 INFO:teuthology.orchestra.run.smithi017.stderr:Setting orchestrator backend to cephadm... 2024-03-26T04:30:28.112 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:27 smithi017 ceph-mon[24814]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:30:28.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:27 smithi017 ceph-mon[24814]: mgrmap e7: y(active, since 1.03224s) 2024-03-26T04:30:28.113 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:27 smithi017 ceph-mon[24814]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:30:28] ENGINE Bus STARTING 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:30:28] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:30:28] ENGINE Bus STARTED 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-mon[24814]: mgrmap e8: y(active, since 2s) 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:29.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:29.032 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:28 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:29.241 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: value unchanged 2024-03-26T04:30:29.770 INFO:teuthology.orchestra.run.smithi017.stderr:Generating ssh key... 2024-03-26T04:30:30.179 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:29 smithi017 ceph-mon[24814]: [26/Mar/2024:04:30:28] ENGINE Bus STARTING 2024-03-26T04:30:30.179 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:29 smithi017 ceph-mon[24814]: [26/Mar/2024:04:30:28] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:30:30.179 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:29 smithi017 ceph-mon[24814]: [26/Mar/2024:04:30:28] ENGINE Bus STARTED 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: Generating public/private rsa key pair. 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: Your identification has been saved in /tmp/tmpkewfdhf7/key. 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: Your public key has been saved in /tmp/tmpkewfdhf7/key.pub. 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: The key fingerprint is: 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: SHA256:ypLAjjYuJzTBIFLzrIrWmIeTdpu7q2j0ClMsI8igd3I ceph-67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: The key's randomart image is: 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: +---[RSA 3072]----+ 2024-03-26T04:30:30.866 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: | .o | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |+ + | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |* o | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |== . | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |*.X E S | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |o%*= o . | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |O@++o o | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |Xo*.o. | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: |+*o*+ | 2024-03-26T04:30:30.867 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: +----[SHA256]-----+ 2024-03-26T04:30:31.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-mon[24814]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:31.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:31.149 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:30 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:31.769 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3LmHySGM9l738/KvUgouMHMyPhHkfhuxnRHUFZPx7xrGv7LrGocLN9DNd92uI8EHQxpD3Ao6OZncXXoCiQgvr+vvI+sNbHUKwkrOtFQGPOA6654umKJV5CTfEnq4OFdXNSekFc8YzU8+6mZrYB3AdBW+D2zcFuu7OR0vtdESW0xYERb7aHzCtNpu1skaZ1dRKIumsPb9SNlUlbNZCL3Szed5M7Y3OiArKvFlDhLO2Sg6M9mQnlJ2D8UnA4aSNbzEny/QSOe8e4AG71XG1VbziuQHLajZn3BORIc6/qXdYHfsZGTR6Ax80TKfNeSSLp0DZGdwTIz8sGH5KWkulZ0WtldKJ2Q8GDMJank1bw46XhTr/xl116wSuRlZO7lm2cj6RYigSJKKLonj3/ep8j2mAy0ssY216rOCGwAcdOsHNRf104tSfgvgC5IbLmUoqy5DMeAsMEG8e0ox1KG03xRAMXaS7CZSemfXoxSDVT8zdkn6UumhRGhCnii0cjp5JfxM= ceph-67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:32.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:31 smithi017 ceph-mon[24814]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:32.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:31 smithi017 ceph-mon[24814]: Generating ssh key... 2024-03-26T04:30:32.307 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:30:32.308 INFO:teuthology.orchestra.run.smithi017.stderr:Adding key to root@localhost authorized_keys... 2024-03-26T04:30:32.308 INFO:teuthology.orchestra.run.smithi017.stderr:Adding host smithi017... 2024-03-26T04:30:33.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:32 smithi017 ceph-mon[24814]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:33.905 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Added host 'smithi017' with addr '172.21.15.17' 2024-03-26T04:30:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:33 smithi017 ceph-mon[24814]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:33 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:34.490 INFO:teuthology.orchestra.run.smithi017.stderr:Deploying unmanaged mon service... 2024-03-26T04:30:35.088 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-26T04:30:35.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:34 smithi017 ceph-mon[24814]: Deploying cephadm binary to smithi017 2024-03-26T04:30:35.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:34 smithi017 ceph-mon[24814]: Added host smithi017 2024-03-26T04:30:35.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:34 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:35.832 INFO:teuthology.orchestra.run.smithi017.stderr:Deploying unmanaged mgr service... 2024-03-26T04:30:36.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:36 smithi017 ceph-mon[24814]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:36.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:36 smithi017 ceph-mon[24814]: Saving service mon spec with placement count:5 2024-03-26T04:30:36.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:36 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:37.297 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-26T04:30:37.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:37 smithi017 ceph-mon[24814]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:37.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:37 smithi017 ceph-mon[24814]: Saving service mgr spec with placement count:2 2024-03-26T04:30:37.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:37 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:37.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:37 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:37.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:37 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:39.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:39 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2755704379' entity='client.admin' 2024-03-26T04:30:40.445 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling the dashboard module... 2024-03-26T04:30:41.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:40 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2801511104' entity='client.admin' 2024-03-26T04:30:41.002 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:40 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:42.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:41 smithi017 ceph-mon[24814]: from='mgr.14120 172.21.15.17:0/852384486' entity='mgr.y' 2024-03-26T04:30:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:41 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3243446887' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T04:30:42.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:41 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setuser ceph since I am not root 2024-03-26T04:30:42.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:41 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setgroup ceph since I am not root 2024-03-26T04:30:42.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:42 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:42.442+0000 7f16e2098000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:30:42.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:42 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:42.534+0000 7f16e2098000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:30:43.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:42 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3243446887' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T04:30:43.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:42 smithi017 ceph-mon[24814]: mgrmap e9: y(active, since 16s) 2024-03-26T04:30:43.457 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.147+0000 7f16e2098000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:30:43.457 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.240+0000 7f16e2098000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:30:43.457 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.327+0000 7f16e2098000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:30:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-26T04:30:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-03-26T04:30:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-26T04:30:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-26T04:30:43.480 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:43.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.478+0000 7f16e2098000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:30:43.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.564+0000 7f16e2098000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:30:43.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.675+0000 7f16e2098000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:30:43.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.761+0000 7f16e2098000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:30:44.015 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3739348429' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:30:44.015 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:43.926+0000 7f16e2098000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:30:44.100 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for the mgr to restart... 2024-03-26T04:30:44.100 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr epoch 9... 2024-03-26T04:30:44.342 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:44.015+0000 7f16e2098000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:30:45.397 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:45.070+0000 7f16e2098000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:30:45.397 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:45.202+0000 7f16e2098000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:30:45.397 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:45.396+0000 7f16e2098000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:30:45.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:45.485+0000 7f16e2098000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:30:45.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:45.623+0000 7f16e2098000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:30:46.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:45.943+0000 7f16e2098000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:30:46.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:46.169+0000 7f16e2098000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:30:46.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:46.256+0000 7f16e2098000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:30:46.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:46.430+0000 7f16e2098000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:30:47.663 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:47 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:47.361+0000 7f16e2098000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:30:47.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:47 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:47.662+0000 7f16e2098000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:30:47.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:47 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:47.759+0000 7f16e2098000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:30:47.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:47 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:30:47.911+0000 7f16e2098000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:30:48.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:30:48] ENGINE Bus STARTING 2024-03-26T04:30:48.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:30:48] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:30:48.987 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:30:48] ENGINE Bus STARTED 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: Active manager daemon y restarted 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: Activating manager daemon y 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: mgrmap e10: y(active, starting, since 0.0529586s) 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: Manager daemon y is now available 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:30:48.988 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:30:48.989 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:30:48.989 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-03-26T04:30:48.989 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-26T04:30:48.989 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "initialized": true 2024-03-26T04:30:48.989 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-03-26T04:30:49.760 INFO:teuthology.orchestra.run.smithi017.stderr:mgr epoch 9 is available 2024-03-26T04:30:49.760 INFO:teuthology.orchestra.run.smithi017.stderr:Generating a dashboard self-signed certificate... 2024-03-26T04:30:50.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:49 smithi017 ceph-mon[24814]: [26/Mar/2024:04:30:48] ENGINE Bus STARTING 2024-03-26T04:30:50.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:49 smithi017 ceph-mon[24814]: [26/Mar/2024:04:30:48] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:30:50.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:49 smithi017 ceph-mon[24814]: [26/Mar/2024:04:30:48] ENGINE Bus STARTED 2024-03-26T04:30:50.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:49 smithi017 ceph-mon[24814]: mgrmap e11: y(active, since 1.05528s) 2024-03-26T04:30:50.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:49 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:30:50.553 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-26T04:30:51.161 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial admin user... 2024-03-26T04:30:51.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:50 smithi017 ceph-mon[24814]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:30:51.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:50 smithi017 ceph-mon[24814]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:30:51.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:50 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:30:51.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:50 smithi017 ceph-mon[24814]: mgrmap e12: y(active, since 2s) 2024-03-26T04:30:51.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:50 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:30:51.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:50 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:30:52.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:51 smithi017 ceph-mon[24814]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:52.470 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$OWCPn7Kug/5k9qdDmX7JVOBfcultJ8BcwvdFjeUkUx3iEcy8Ijo6W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711427452, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T04:30:53.014 INFO:teuthology.orchestra.run.smithi017.stderr:Fetching dashboard port number... 2024-03-26T04:30:53.608 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 8443 2024-03-26T04:30:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:53 smithi017 ceph-mon[24814]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:30:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:53 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:30:54.172 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-03-26T04:30:54.179 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-03-26T04:30:54.180 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-03-26T04:30:54.180 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr:Ceph Dashboard is now available at: 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr: URL: https://smithi017.front.sepia.ceph.com:8443/ 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr: User: admin 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr: Password: mk6gl4tsta 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:54.182 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling autotune for osd_memory_target 2024-03-26T04:30:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:54 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3475426397' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T04:30:55.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:55 smithi017 ceph-mon[24814]: mgrmap e13: y(active, since 6s) 2024-03-26T04:30:56.151 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-26T04:30:56.710 INFO:teuthology.orchestra.run.smithi017.stderr:You can access the Ceph CLI with: 2024-03-26T04:30:56.711 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:56.711 INFO:teuthology.orchestra.run.smithi017.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:30:56.711 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:56.711 INFO:teuthology.orchestra.run.smithi017.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-26T04:30:56.711 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:56.711 INFO:teuthology.orchestra.run.smithi017.stderr: ceph telemetry on 2024-03-26T04:30:56.712 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:56.712 INFO:teuthology.orchestra.run.smithi017.stderr:For more information see: 2024-03-26T04:30:56.712 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:56.712 INFO:teuthology.orchestra.run.smithi017.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-26T04:30:56.712 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:30:56.713 INFO:teuthology.orchestra.run.smithi017.stderr:Bootstrap complete. 2024-03-26T04:30:56.751 INFO:tasks.cephadm:Fetching config... 2024-03-26T04:30:56.751 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:30:56.751 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T04:30:56.768 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T04:30:56.768 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:30:56.769 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T04:30:56.824 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T04:30:56.824 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:30:56.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T04:30:56.891 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T04:30:56.892 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:30:56.892 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T04:30:56.947 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T04:30:56.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3LmHySGM9l738/KvUgouMHMyPhHkfhuxnRHUFZPx7xrGv7LrGocLN9DNd92uI8EHQxpD3Ao6OZncXXoCiQgvr+vvI+sNbHUKwkrOtFQGPOA6654umKJV5CTfEnq4OFdXNSekFc8YzU8+6mZrYB3AdBW+D2zcFuu7OR0vtdESW0xYERb7aHzCtNpu1skaZ1dRKIumsPb9SNlUlbNZCL3Szed5M7Y3OiArKvFlDhLO2Sg6M9mQnlJ2D8UnA4aSNbzEny/QSOe8e4AG71XG1VbziuQHLajZn3BORIc6/qXdYHfsZGTR6Ax80TKfNeSSLp0DZGdwTIz8sGH5KWkulZ0WtldKJ2Q8GDMJank1bw46XhTr/xl116wSuRlZO7lm2cj6RYigSJKKLonj3/ep8j2mAy0ssY216rOCGwAcdOsHNRf104tSfgvgC5IbLmUoqy5DMeAsMEG8e0ox1KG03xRAMXaS7CZSemfXoxSDVT8zdkn6UumhRGhCnii0cjp5JfxM= ceph-67a15696-eb29-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:30:57.029 INFO:teuthology.orchestra.run.smithi017.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3LmHySGM9l738/KvUgouMHMyPhHkfhuxnRHUFZPx7xrGv7LrGocLN9DNd92uI8EHQxpD3Ao6OZncXXoCiQgvr+vvI+sNbHUKwkrOtFQGPOA6654umKJV5CTfEnq4OFdXNSekFc8YzU8+6mZrYB3AdBW+D2zcFuu7OR0vtdESW0xYERb7aHzCtNpu1skaZ1dRKIumsPb9SNlUlbNZCL3Szed5M7Y3OiArKvFlDhLO2Sg6M9mQnlJ2D8UnA4aSNbzEny/QSOe8e4AG71XG1VbziuQHLajZn3BORIc6/qXdYHfsZGTR6Ax80TKfNeSSLp0DZGdwTIz8sGH5KWkulZ0WtldKJ2Q8GDMJank1bw46XhTr/xl116wSuRlZO7lm2cj6RYigSJKKLonj3/ep8j2mAy0ssY216rOCGwAcdOsHNRf104tSfgvgC5IbLmUoqy5DMeAsMEG8e0ox1KG03xRAMXaS7CZSemfXoxSDVT8zdkn6UumhRGhCnii0cjp5JfxM= ceph-67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:57.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3LmHySGM9l738/KvUgouMHMyPhHkfhuxnRHUFZPx7xrGv7LrGocLN9DNd92uI8EHQxpD3Ao6OZncXXoCiQgvr+vvI+sNbHUKwkrOtFQGPOA6654umKJV5CTfEnq4OFdXNSekFc8YzU8+6mZrYB3AdBW+D2zcFuu7OR0vtdESW0xYERb7aHzCtNpu1skaZ1dRKIumsPb9SNlUlbNZCL3Szed5M7Y3OiArKvFlDhLO2Sg6M9mQnlJ2D8UnA4aSNbzEny/QSOe8e4AG71XG1VbziuQHLajZn3BORIc6/qXdYHfsZGTR6Ax80TKfNeSSLp0DZGdwTIz8sGH5KWkulZ0WtldKJ2Q8GDMJank1bw46XhTr/xl116wSuRlZO7lm2cj6RYigSJKKLonj3/ep8j2mAy0ssY216rOCGwAcdOsHNRf104tSfgvgC5IbLmUoqy5DMeAsMEG8e0ox1KG03xRAMXaS7CZSemfXoxSDVT8zdkn6UumhRGhCnii0cjp5JfxM= ceph-67a15696-eb29-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:30:57.089 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3LmHySGM9l738/KvUgouMHMyPhHkfhuxnRHUFZPx7xrGv7LrGocLN9DNd92uI8EHQxpD3Ao6OZncXXoCiQgvr+vvI+sNbHUKwkrOtFQGPOA6654umKJV5CTfEnq4OFdXNSekFc8YzU8+6mZrYB3AdBW+D2zcFuu7OR0vtdESW0xYERb7aHzCtNpu1skaZ1dRKIumsPb9SNlUlbNZCL3Szed5M7Y3OiArKvFlDhLO2Sg6M9mQnlJ2D8UnA4aSNbzEny/QSOe8e4AG71XG1VbziuQHLajZn3BORIc6/qXdYHfsZGTR6Ax80TKfNeSSLp0DZGdwTIz8sGH5KWkulZ0WtldKJ2Q8GDMJank1bw46XhTr/xl116wSuRlZO7lm2cj6RYigSJKKLonj3/ep8j2mAy0ssY216rOCGwAcdOsHNRf104tSfgvgC5IbLmUoqy5DMeAsMEG8e0ox1KG03xRAMXaS7CZSemfXoxSDVT8zdkn6UumhRGhCnii0cjp5JfxM= ceph-67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:30:57.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T04:30:57.418 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:57 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2200682911' entity='client.admin' 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/768162198' entity='client.admin' 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:00.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:30:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:00.438 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T04:31:00.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T04:31:03.218 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2024-03-26T04:31:03.218 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:31:03.218 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:31:03.236 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:31:03.236 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:31:03.293 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2024-03-26T04:31:03.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch host add smithi129 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:31:03.764 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:31:03.765 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:03.765 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:06.067 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi129' with addr '172.21.15.129' 2024-03-26T04:31:06.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:05 smithi017 ceph-mon[24814]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:06.538 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:31:07.133 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: Deploying cephadm binary to smithi129 2024-03-26T04:31:07.133 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:07.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:07.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:07.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:07.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:07.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:07 smithi017 ceph-mon[24814]: Added host smithi129 2024-03-26T04:31:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:07 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:08.317 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:31:08.318 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}] 2024-03-26T04:31:08.929 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T04:31:08.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T04:31:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:08 smithi017 ceph-mon[24814]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:08 smithi017 ceph-mon[24814]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:31:11.586 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-03-26T04:31:11.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:11 smithi017 ceph-mon[24814]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:11.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:11.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:11.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:31:11.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:11.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:11 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2955819859' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:31:12.192 INFO:tasks.cephadm:Adding mon.a on smithi017 2024-03-26T04:31:12.192 INFO:tasks.cephadm:Adding mon.c on smithi017 2024-03-26T04:31:12.192 INFO:tasks.cephadm:Adding mon.b on smithi129 2024-03-26T04:31:12.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi129:172.21.15.129=b' 2024-03-26T04:31:12.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:12 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2955819859' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:31:12.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:12 smithi017 ceph-mon[24814]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:13.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:13 smithi017 ceph-mon[24814]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:13.988 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mon update... 2024-03-26T04:31:14.659 DEBUG:teuthology.orchestra.run.smithi017:mon.c> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T04:31:14.662 DEBUG:teuthology.orchestra.run.smithi129:mon.b> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T04:31:14.664 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:31:14.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:31:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi129:172.21.15.129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi129:172.21.15.129=b;count:3 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:14 smithi017 ceph-mon[24814]: Deploying daemon mon.c on smithi017 2024-03-26T04:31:16.523 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:31:16.523 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","modified":"2024-03-26T04:29:56.678288Z","created":"2024-03-26T04:29:56.678288Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T04:31:16.525 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 1 2024-03-26T04:31:17.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:16 smithi017 ceph-mon[24814]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:17.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:16 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/3146400877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:31:18.098 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:31:18.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:31:23.034 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:31:23.034 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","modified":"2024-03-26T04:31:17.650167Z","created":"2024-03-26T04:29:56.678288Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:31:23.036 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 2 2024-03-26T04:31:23.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: mon.a calling monitor election 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: mon.c calling monitor election 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:31:23.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: fsmap 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: mgrmap e13: y(active, since 34s) 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: overall HEALTH_OK 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:23.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:24.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[24814]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:24.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[24814]: Deploying daemon mon.b on smithi129 2024-03-26T04:31:24.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/961576888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:31:24.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:24.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[29581]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:24.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[29581]: Deploying daemon mon.b on smithi129 2024-03-26T04:31:24.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/961576888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:31:24.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:25.089 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:31:25.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:31:26.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:25 smithi017 ceph-mon[24814]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:26.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:25 smithi017 ceph-mon[29581]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:27.241 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:31:27.241 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","modified":"2024-03-26T04:31:17.650167Z","created":"2024-03-26T04:29:56.678288Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:31:27.243 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 2 2024-03-26T04:31:27.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:27 smithi129 ceph-mon[25657]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-26T04:31:27.314 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:27 smithi129 bash[25618]: 685e2a3e6efc0a0111f57508765e00489fd8a18d9fa5e0efb3de7c384dcb6509 2024-03-26T04:31:27.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:27 smithi129 systemd[1]: Started Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:31:28.840 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:31:28.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: mon.c calling monitor election 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: mon.a calling monitor election 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:32.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: fsmap 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: mgrmap e13: y(active, since 44s) 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:31:32.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: mon.c calling monitor election 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: mon.a calling monitor election 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: fsmap 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: mgrmap e13: y(active, since 44s) 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:32.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:32.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:33.553 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-26T04:31:33.553 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","modified":"2024-03-26T04:31:27.290067Z","created":"2024-03-26T04:29:56.678288Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:31:33.554 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 3 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/4100466825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:33.651 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.652 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.652 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:33.652 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.652 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:33.652 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:33 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/4100466825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:31:34.157 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T04:31:34.157 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T04:31:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.c calling monitor election 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.a calling monitor election 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:34.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: fsmap 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mgrmap e13: y(active, since 44s) 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:34.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:34.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:34.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/4100466825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: mon.b calling monitor election 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: mon.b calling monitor election 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: mon.c calling monitor election 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: mon.a calling monitor election 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: fsmap 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: mgrmap e13: y(active, since 46s) 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: Cluster is now healthy 2024-03-26T04:31:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[29581]: overall HEALTH_OK 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: mon.b calling monitor election 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: mon.b calling monitor election 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: mon.c calling monitor election 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: mon.a calling monitor election 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: fsmap 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: mgrmap e13: y(active, since 46s) 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: Cluster is now healthy 2024-03-26T04:31:34.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:34 smithi017 ceph-mon[24814]: overall HEALTH_OK 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.b calling monitor election 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.b calling monitor election 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.c calling monitor election 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.a calling monitor election 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: fsmap 2024-03-26T04:31:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:31:35.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: mgrmap e13: y(active, since 46s) 2024-03-26T04:31:35.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:31:35.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: Cluster is now healthy 2024-03-26T04:31:35.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:34 smithi129 ceph-mon[25657]: overall HEALTH_OK 2024-03-26T04:31:36.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:35 smithi017 ceph-mon[24814]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:36.060 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:35 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:36.061 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:35 smithi017 ceph-mon[29581]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:36.061 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:35 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:36.116 INFO:teuthology.orchestra.run.smithi017.stdout:# minimal ceph.conf for 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:31:36.117 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-03-26T04:31:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:31:36.117 INFO:teuthology.orchestra.run.smithi017.stdout: mon_host = [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-03-26T04:31:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:35 smithi129 ceph-mon[25657]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:35 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:31:36.979 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T04:31:36.979 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:31:36.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:31:37.007 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:31:37.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:31:37.076 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:31:37.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:31:37.109 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:31:37.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:31:37.178 INFO:tasks.cephadm:Adding mgr.y on smithi017 2024-03-26T04:31:37.179 INFO:tasks.cephadm:Adding mgr.x on smithi129 2024-03-26T04:31:37.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi017=y;smithi129=x' 2024-03-26T04:31:37.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/935184407' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:37.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:37.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:37.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:31:37.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:37.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/935184407' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:37.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:37.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:37.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:31:37.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:36 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:37.244 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:36 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/935184407' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:37.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:36 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:37.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:36 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:37.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:36 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:31:37.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:36 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:38.116 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:37 smithi129 ceph-mon[25657]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:38.116 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:37 smithi129 ceph-mon[25657]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:31:38.116 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:37 smithi129 ceph-mon[25657]: Reconfiguring daemon mon.c on smithi017 2024-03-26T04:31:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:37 smithi017 ceph-mon[29581]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:37 smithi017 ceph-mon[29581]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:31:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:37 smithi017 ceph-mon[29581]: Reconfiguring daemon mon.c on smithi017 2024-03-26T04:31:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:37 smithi017 ceph-mon[24814]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:37 smithi017 ceph-mon[24814]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:31:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:37 smithi017 ceph-mon[24814]: Reconfiguring daemon mon.c on smithi017 2024-03-26T04:31:38.934 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mgr update... 2024-03-26T04:31:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:31:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:39.276 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:31:39.276 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:39.276 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: Reconfiguring daemon mon.b on smithi129 2024-03-26T04:31:39.276 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:39.276 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: Saving service mgr spec with placement smithi017=y;smithi129=x;count:2 2024-03-26T04:31:39.276 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:39.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:39.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:39.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: Reconfiguring daemon mon.b on smithi129 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: Saving service mgr spec with placement smithi017=y;smithi129=x;count:2 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:31:39.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: Reconfiguring daemon mon.b on smithi129 2024-03-26T04:31:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: Saving service mgr spec with placement smithi017=y;smithi129=x;count:2 2024-03-26T04:31:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:40.151 DEBUG:teuthology.orchestra.run.smithi129:mgr.x> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T04:31:40.195 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T04:31:40.196 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:31:40.196 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:31:40.215 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:31:40.215 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-03-26T04:31:40.272 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-26 04:31:17.159500586 +0000 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-26 04:28:50.118888567 +0000 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-26 04:28:50.118888567 +0000 2024-03-26T04:31:40.273 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-26 04:28:50.118888567 +0000 2024-03-26T04:31:40.273 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:31:40.343 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-26T04:31:40.343 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-26T04:31:40.344 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000123319 s, 4.2 MB/s 2024-03-26T04:31:40.345 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:31:40.403 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-03-26T04:31:40.460 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-26 04:31:17.159500586 +0000 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-26 04:28:50.437885266 +0000 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-26 04:28:50.437885266 +0000 2024-03-26T04:31:40.461 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-26 04:28:50.437885266 +0000 2024-03-26T04:31:40.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:31:40.530 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-26T04:31:40.531 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-26T04:31:40.531 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000122262 s, 4.2 MB/s 2024-03-26T04:31:40.532 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:31:40.589 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-26 04:31:17.159500586 +0000 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-26 04:28:50.760881923 +0000 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-26 04:28:50.760881923 +0000 2024-03-26T04:31:40.647 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-26 04:28:50.760881923 +0000 2024-03-26T04:31:40.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:31:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-26T04:31:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-26T04:31:40.715 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000119856 s, 4.3 MB/s 2024-03-26T04:31:40.717 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:31:40.775 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-03-26T04:31:40.831 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-03-26 04:31:17.160500575 +0000 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-03-26 04:28:51.099878415 +0000 2024-03-26T04:31:40.832 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-03-26 04:28:51.099878415 +0000 2024-03-26T04:31:40.833 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-03-26 04:28:51.099878415 +0000 2024-03-26T04:31:40.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:31:40.898 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-03-26T04:31:40.899 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-03-26T04:31:40.899 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000111194 s, 4.6 MB/s 2024-03-26T04:31:40.900 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:31:40.957 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:31:40.957 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:31:40.975 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:31:40.975 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-03-26T04:31:41.031 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:31:41.031 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:41.031 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-26T04:31:41.031 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:41.031 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:41.031 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-26 04:31:26.292448249 +0000 2024-03-26T04:31:41.032 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-26 04:28:48.290286992 +0000 2024-03-26T04:31:41.032 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-26 04:28:48.290286992 +0000 2024-03-26T04:31:41.032 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-26 04:28:48.290286992 +0000 2024-03-26T04:31:41.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:41.096 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:40 smithi129 ceph-mon[25657]: Deploying daemon mgr.x on smithi129 2024-03-26T04:31:41.098 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-26T04:31:41.098 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-26T04:31:41.098 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000220144 s, 2.3 MB/s 2024-03-26T04:31:41.099 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:31:41.155 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-03-26T04:31:41.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:41.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:41.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:41.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:41.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:41.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[29581]: Deploying daemon mgr.x on smithi129 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:31:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:31:41.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:31:41.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:41.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:40 smithi017 ceph-mon[24814]: Deploying daemon mgr.x on smithi129 2024-03-26T04:31:41.211 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:31:41.211 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:41.211 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T04:31:41.211 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:41.212 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:41.212 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-26 04:31:26.293448227 +0000 2024-03-26T04:31:41.212 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-26 04:28:48.631279302 +0000 2024-03-26T04:31:41.212 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-26 04:28:48.631279302 +0000 2024-03-26T04:31:41.212 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-26 04:28:48.631279302 +0000 2024-03-26T04:31:41.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:31:41.279 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-26T04:31:41.280 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-26T04:31:41.280 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000203251 s, 2.5 MB/s 2024-03-26T04:31:41.281 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:31:41.339 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-26 04:31:41.237112214 +0000 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-26 04:28:48.961271859 +0000 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-26 04:28:48.961271859 +0000 2024-03-26T04:31:41.394 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-26 04:28:48.961271859 +0000 2024-03-26T04:31:41.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:31:41.468 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-26T04:31:41.468 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-26T04:31:41.468 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.00019901 s, 2.6 MB/s 2024-03-26T04:31:41.469 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:31:41.526 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-03-26T04:31:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:31:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:31:41.581 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T04:31:41.582 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:31:41.582 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:31:41.582 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-26 04:31:41.410108327 +0000 2024-03-26T04:31:41.582 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-26 04:28:49.301264191 +0000 2024-03-26T04:31:41.582 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-26 04:28:49.301264191 +0000 2024-03-26T04:31:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-26 04:28:49.301264191 +0000 2024-03-26T04:31:41.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:31:41.648 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-26T04:31:41.649 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-26T04:31:41.649 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000202934 s, 2.5 MB/s 2024-03-26T04:31:41.650 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:31:41.705 INFO:tasks.cephadm:Deploying osd.0 on smithi017 with /dev/vg_nvme/lv_4... 2024-03-26T04:31:41.706 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:31:42.023 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:41 smithi017 ceph-mon[24814]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:42.023 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:41 smithi017 ceph-mon[29581]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:42.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:41 smithi129 ceph-mon[25657]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[24814]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[29581]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:43.315 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:43.316 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:43.316 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:43.316 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:43.316 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-mon[25657]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:43.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.277+0000 7f3169b41000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:31:43.684 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:31:43.684 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:31:43.684 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-03-26T04:31:43.684 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-03-26T04:31:43.685 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00828883 s, 1.3 GB/s 2024-03-26T04:31:43.685 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-03-26T04:31:43.695 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.425+0000 7f3169b41000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:31:43.695 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.505+0000 7f3169b41000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:31:43.695 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.613+0000 7f3169b41000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:31:43.695 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.695+0000 7f3169b41000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:31:44.062 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.857+0000 7f3169b41000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:31:44.062 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:43.941+0000 7f3169b41000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:31:44.329 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:44.329 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:44.329 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[29581]: Reconfiguring daemon mgr.y on smithi017 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:44.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:44 smithi017 ceph-mon[24814]: Reconfiguring daemon mgr.y on smithi017 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-mon[25657]: Reconfiguring daemon mgr.y on smithi017 2024-03-26T04:31:44.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_4 2024-03-26T04:31:45.301 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:44.981+0000 7f3169b41000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:31:45.302 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:45.109+0000 7f3169b41000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:31:45.302 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:45.301+0000 7f3169b41000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:31:45.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-mon[25657]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:45.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:45.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:45.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:45.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:45.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:45.646 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:45.387+0000 7f3169b41000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:31:45.646 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:45.521+0000 7f3169b41000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:31:45.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[24814]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[29581]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:45.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:45.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:45.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:31:46.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:45.850+0000 7f3169b41000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:31:46.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:46.066+0000 7f3169b41000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:31:46.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:46.150+0000 7f3169b41000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:31:46.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:46.316+0000 7f3169b41000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:31:47.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[24814]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:47.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[24814]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:47.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:31:47.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:31:47.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:47.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[29581]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:47.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[29581]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:47.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:31:47.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:31:47.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:47.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-mon[25657]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:47.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-mon[25657]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:31:47.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:31:47.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:31:47.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:47.527 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:47.234+0000 7f3169b41000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:31:47.527 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:47.524+0000 7f3169b41000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:31:47.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:47.617+0000 7f3169b41000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:31:47.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:31:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:31:47.765+0000 7f3169b41000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:31:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: Standby manager daemon x started 2024-03-26T04:31:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:31:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: Standby manager daemon x started 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:31:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: Standby manager daemon x started 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/368464601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:31:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:31:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-26T04:31:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:31:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2313108896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]: dispatch 2024-03-26T04:31:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]: dispatch 2024-03-26T04:31:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]': finished 2024-03-26T04:31:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:31:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:49 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:31:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2313108896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]: dispatch 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]: dispatch 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]': finished 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:31:49.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2313108896' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]: dispatch 2024-03-26T04:31:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]: dispatch 2024-03-26T04:31:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55282a5c-04c7-4f56-ae85-1f7b13034dee"}]': finished 2024-03-26T04:31:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:31:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:49 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:31:50.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:50 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1515012899' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:31:50.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:50 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1515012899' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:31:50.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:50 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1515012899' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:31:52.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:51 smithi017 ceph-mon[24814]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:52.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:51 smithi017 ceph-mon[29581]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:51 smithi129 ceph-mon[25657]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:54.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:53 smithi017 ceph-mon[29581]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:54.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:53 smithi017 ceph-mon[24814]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:53 smithi129 ceph-mon[25657]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:56.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:55 smithi017 ceph-mon[24814]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:56.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:55 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:31:56.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:55 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:56.315 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:55 smithi017 ceph-mon[29581]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:56.315 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:55 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:31:56.315 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:55 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:55 smithi129 ceph-mon[25657]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:55 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:31:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:55 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:57.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:56 smithi129 ceph-mon[25657]: Deploying daemon osd.0 on smithi017 2024-03-26T04:31:57.306 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:56 smithi017 ceph-mon[24814]: Deploying daemon osd.0 on smithi017 2024-03-26T04:31:57.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:56 smithi017 ceph-mon[29581]: Deploying daemon osd.0 on smithi017 2024-03-26T04:31:58.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:57 smithi017 ceph-mon[24814]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:58.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:57 smithi017 ceph-mon[29581]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:58.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:57 smithi129 ceph-mon[25657]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:59.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:31:58 smithi129 ceph-mon[25657]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:59.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:31:58 smithi017 ceph-mon[24814]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:31:59.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:31:58 smithi017 ceph-mon[29581]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:01.596 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[29581]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:01.596 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:01.596 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[24814]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:01.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:01 smithi129 ceph-mon[25657]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:01.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:01.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:03.263 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 0 on host 'smithi017' 2024-03-26T04:32:03.567 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:03 smithi017 ceph-mon[29581]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:03.567 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:03 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:03.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:03 smithi017 ceph-mon[24814]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:03.568 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:03 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:03.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:03 smithi129 ceph-mon[25657]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:03.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:03 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:03.902 DEBUG:teuthology.orchestra.run.smithi017:osd.0> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T04:32:03.905 INFO:tasks.cephadm:Deploying osd.1 on smithi017 with /dev/vg_nvme/lv_3... 2024-03-26T04:32:03.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:32:04.545 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:04 smithi017 ceph-mon[24814]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:32:04.546 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:04 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:04.546 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:04 smithi017 ceph-mon[24814]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:04.546 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:04 smithi017 ceph-mon[29581]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:32:04.546 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:04 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:04.546 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:04 smithi017 ceph-mon[29581]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:04.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:04 smithi129 ceph-mon[25657]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:32:04.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:04 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:04.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:04 smithi129 ceph-mon[25657]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:05.527 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[24814]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:32:05.527 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[24814]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:32:05.527 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:05.528 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[24814]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:05.528 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[29581]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:32:05.528 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[29581]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:32:05.528 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:05.528 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-mon[29581]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:05.528 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[32428]: 2024-03-26T04:32:05.310+0000 7f1f989da700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T04:32:05.528 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:05 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[32428]: 2024-03-26T04:32:05.312+0000 7f1f91b6e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:32:05.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:05 smithi129 ceph-mon[25657]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:32:05.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:05 smithi129 ceph-mon[25657]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:32:05.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:05 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:05.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:05 smithi129 ceph-mon[25657]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:05.924 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:32:05.925 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:32:05.925 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-03-26T04:32:05.925 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-03-26T04:32:05.925 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00919422 s, 1.1 GB/s 2024-03-26T04:32:05.925 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-03-26T04:32:06.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:06 smithi129 ceph-mon[25657]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:06.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:06 smithi129 ceph-mon[25657]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:32:06.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:06 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:06.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:06 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:06.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[24814]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[24814]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[29581]: from='osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[29581]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:06 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:32:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:32:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498] boot 2024-03-26T04:32:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498] boot 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:32:07.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:07 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:07.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_3 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: osd.0 [v2:172.21.15.17:6802/857707498,v1:172.21.15.17:6803/857707498] boot 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:32:07.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:07 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:32:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:08 smithi017 ceph-mon[24814]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:32:08.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:08 smithi017 ceph-mon[29581]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:32:08.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:08 smithi129 ceph-mon[25657]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:09.389 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:09 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:09.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:09.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:09.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:09.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:09.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:09.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:09 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:10 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi017 2024-03-26T04:32:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:10 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi017 to 16069M 2024-03-26T04:32:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:10 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:10 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:10.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:10 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:10.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:10 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:10.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi017 2024-03-26T04:32:10.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi017 to 16069M 2024-03-26T04:32:10.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:10.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi017 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi017 to 16069M 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:10.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:10 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:11 smithi129 ceph-mon[25657]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:11 smithi129 ceph-mon[25657]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:12.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:11 smithi017 ceph-mon[24814]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:12.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:11 smithi017 ceph-mon[24814]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:12.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:11 smithi017 ceph-mon[29581]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:12.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:11 smithi017 ceph-mon[29581]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:13.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:12 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/281719014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]: dispatch 2024-03-26T04:32:13.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:12 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]: dispatch 2024-03-26T04:32:13.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:12 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]': finished 2024-03-26T04:32:13.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:12 smithi129 ceph-mon[25657]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:32:13.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:12 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:13.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:12 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3475190297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:32:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/281719014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]: dispatch 2024-03-26T04:32:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]: dispatch 2024-03-26T04:32:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]': finished 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[24814]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3475190297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/281719014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]: dispatch 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]: dispatch 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b8f13d8-440b-4859-adac-d5dba48d4677"}]': finished 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[29581]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:12 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3475190297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:32:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:13 smithi129 ceph-mon[25657]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:13 smithi017 ceph-mon[24814]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:13 smithi017 ceph-mon[29581]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:16.277 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:15 smithi017 ceph-mon[24814]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:16.277 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:15 smithi017 ceph-mon[29581]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:15 smithi129 ceph-mon[25657]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:17 smithi017 ceph-mon[24814]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:17 smithi017 ceph-mon[29581]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:17 smithi129 ceph-mon[25657]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:19.312 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:18 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:32:19.313 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:18 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:19.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:18 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:32:19.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:18 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:18 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:32:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:18 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:20.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:19 smithi017 ceph-mon[24814]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:20.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:19 smithi017 ceph-mon[24814]: Deploying daemon osd.1 on smithi017 2024-03-26T04:32:20.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:19 smithi017 ceph-mon[29581]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:20.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:19 smithi017 ceph-mon[29581]: Deploying daemon osd.1 on smithi017 2024-03-26T04:32:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:19 smithi129 ceph-mon[25657]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:19 smithi129 ceph-mon[25657]: Deploying daemon osd.1 on smithi017 2024-03-26T04:32:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:21 smithi129 ceph-mon[25657]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:22.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:21 smithi017 ceph-mon[24814]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:22.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:21 smithi017 ceph-mon[29581]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:24.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[24814]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:24.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:24.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:24.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:24.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:24.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[29581]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:24.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:24.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:24.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:24.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:23 smithi129 ceph-mon[25657]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:26.234 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:25 smithi017 ceph-mon[29581]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:26.234 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:25 smithi017 ceph-mon[24814]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:26.292 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 1 on host 'smithi017' 2024-03-26T04:32:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:25 smithi129 ceph-mon[25657]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:26.889 DEBUG:teuthology.orchestra.run.smithi017:osd.1> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T04:32:26.892 INFO:tasks.cephadm:Deploying osd.2 on smithi017 with /dev/vg_nvme/lv_2... 2024-03-26T04:32:26.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:32:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:26 smithi017 ceph-mon[24814]: from='osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:32:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:26 smithi017 ceph-mon[24814]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:32:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:27.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:26 smithi017 ceph-mon[29581]: from='osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:32:27.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:26 smithi017 ceph-mon[29581]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:32:27.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:27.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:26 smithi129 ceph-mon[25657]: from='osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:32:27.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:26 smithi129 ceph-mon[25657]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:32:27.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:32:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:32:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: from='osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: from='osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:28.207 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[35408]: 2024-03-26T04:32:27.988+0000 7fa86c255700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T04:32:28.207 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:27 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[35408]: 2024-03-26T04:32:27.990+0000 7fa866bec700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:32:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:32:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:32:28.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:28.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: from='osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:28.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:28.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:27 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:29.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[29581]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:29.261 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[29581]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:32:29.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:29.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:29.262 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[24814]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:29.262 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[24814]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:32:29.262 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:29.262 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:28 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:28 smithi129 ceph-mon[25657]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:28 smithi129 ceph-mon[25657]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:32:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:28 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:28 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[24814]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[24814]: osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074] boot 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[24814]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:32:30.127 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:32:30.128 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[29581]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:30.128 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[29581]: osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074] boot 2024-03-26T04:32:30.128 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[29581]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:32:30.128 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:29 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:30.252 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:32:30.252 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:32:30.252 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-03-26T04:32:30.252 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-03-26T04:32:30.252 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125046 s, 839 MB/s 2024-03-26T04:32:30.252 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-03-26T04:32:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:29 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:32:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:29 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:32:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:29 smithi129 ceph-mon[25657]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:32:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:29 smithi129 ceph-mon[25657]: osd.1 [v2:172.21.15.17:6810/3878069074,v1:172.21.15.17:6811/3878069074] boot 2024-03-26T04:32:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:29 smithi129 ceph-mon[25657]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:32:30.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:29 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:32:30.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_2 2024-03-26T04:32:31.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:30 smithi017 ceph-mon[24814]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:31.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:30 smithi017 ceph-mon[24814]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:32:31.100 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:30 smithi017 ceph-mon[29581]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:31.100 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:30 smithi017 ceph-mon[29581]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:32:31.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:30 smithi129 ceph-mon[25657]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:31.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:30 smithi129 ceph-mon[25657]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi017 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi017 to 8034M 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:32.635 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi017 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi017 to 8034M 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:32.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:32 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi017 2024-03-26T04:32:32.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi017 to 8034M 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:32.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:32 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:33.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[24814]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[29581]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:33 smithi129 ceph-mon[25657]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:33 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:33 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:33 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:34.537 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:34 smithi017 ceph-mon[24814]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:34.537 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:34 smithi017 ceph-mon[29581]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:34 smithi129 ceph-mon[25657]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:35 smithi129 ceph-mon[25657]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:35 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1841401412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbff43d2-2aec-401f-897d-afc6b7ab595b"}]: dispatch 2024-03-26T04:32:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:35 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1841401412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbff43d2-2aec-401f-897d-afc6b7ab595b"}]': finished 2024-03-26T04:32:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:35 smithi129 ceph-mon[25657]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:32:36.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:35 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:36.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:35 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1549678806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[24814]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1841401412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbff43d2-2aec-401f-897d-afc6b7ab595b"}]: dispatch 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1841401412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbff43d2-2aec-401f-897d-afc6b7ab595b"}]': finished 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[24814]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1549678806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[29581]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1841401412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbff43d2-2aec-401f-897d-afc6b7ab595b"}]: dispatch 2024-03-26T04:32:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1841401412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbff43d2-2aec-401f-897d-afc6b7ab595b"}]': finished 2024-03-26T04:32:36.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[29581]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:32:36.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:36.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:35 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1549678806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:32:38.257 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:37 smithi017 ceph-mon[24814]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:38.258 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:37 smithi017 ceph-mon[29581]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:38.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:37 smithi129 ceph-mon[25657]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:40.164 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:39 smithi017 ceph-mon[24814]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:40.165 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:39 smithi017 ceph-mon[29581]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:39 smithi129 ceph-mon[25657]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:42.104 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:41 smithi017 ceph-mon[24814]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:42.104 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:41 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:32:42.104 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:41 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:42.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:41 smithi017 ceph-mon[29581]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:42.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:41 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:32:42.105 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:41 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:41 smithi129 ceph-mon[25657]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:41 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:32:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:41 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:43.088 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:42 smithi017 ceph-mon[24814]: Deploying daemon osd.2 on smithi017 2024-03-26T04:32:43.089 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:42 smithi017 ceph-mon[29581]: Deploying daemon osd.2 on smithi017 2024-03-26T04:32:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:42 smithi129 ceph-mon[25657]: Deploying daemon osd.2 on smithi017 2024-03-26T04:32:44.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:43 smithi017 ceph-mon[24814]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:44.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:43 smithi017 ceph-mon[29581]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:43 smithi129 ceph-mon[25657]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:46.146 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:45 smithi017 ceph-mon[24814]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:46.147 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:45 smithi017 ceph-mon[29581]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:45 smithi129 ceph-mon[25657]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:47.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:47 smithi129 ceph-mon[25657]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:47.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:47.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:47.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:47.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:47 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:47.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[24814]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:47.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:47.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:47.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:47.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:47.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[29581]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:47.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:47.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:47.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:47.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:47 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:48.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:32:48.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:32:48.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:32:48.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:32:48.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:32:48.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:32:49.227 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 2 on host 'smithi017' 2024-03-26T04:32:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:49 smithi017 ceph-mon[24814]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:49 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:49.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:49 smithi017 ceph-mon[29581]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:49.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:49 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:49.839 DEBUG:teuthology.orchestra.run.smithi017:osd.2> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:32:49.842 INFO:tasks.cephadm:Deploying osd.3 on smithi017 with /dev/vg_nvme/lv_1... 2024-03-26T04:32:49.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:32:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:49 smithi129 ceph-mon[25657]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:49 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:50.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:50 smithi017 ceph-mon[24814]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:32:50.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:50 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:50.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:50 smithi017 ceph-mon[29581]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:32:50.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:50 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:50 smithi129 ceph-mon[25657]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:32:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:50 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[24814]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[24814]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:32:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[24814]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:32:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[24814]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[29581]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[29581]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:32:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[29581]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:32:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-mon[29581]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:51.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[38319]: 2024-03-26T04:32:51.409+0000 7f7f2ec2a700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T04:32:51.707 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[38319]: 2024-03-26T04:32:51.413+0000 7f7f28dc0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:32:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:51 smithi129 ceph-mon[25657]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:51.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:51 smithi129 ceph-mon[25657]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:32:51.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:51 smithi129 ceph-mon[25657]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:32:51.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:51 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:51.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:51 smithi129 ceph-mon[25657]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[24814]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[24814]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[29581]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[29581]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:52 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:52 smithi129 ceph-mon[25657]: from='osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:32:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:52 smithi129 ceph-mon[25657]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:32:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:52 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:52 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:53.247 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:32:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:32:53.248 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-03-26T04:32:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-03-26T04:32:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00919628 s, 1.1 GB/s 2024-03-26T04:32:53.248 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-03-26T04:32:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:32:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[24814]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[24814]: osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605] boot 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[24814]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[29581]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[29581]: osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605] boot 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[29581]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:32:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:53 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:53.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:53 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:32:53.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:53 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:32:53.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:53 smithi129 ceph-mon[25657]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:32:53.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:53 smithi129 ceph-mon[25657]: osd.2 [v2:172.21.15.17:6818/24179605,v1:172.21.15.17:6819/24179605] boot 2024-03-26T04:32:53.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:53 smithi129 ceph-mon[25657]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:32:53.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:53 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:32:53.905 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_1 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:54.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:54.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:54.679 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:32:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T04:32:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:32:54.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:54.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:54.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:54.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:54.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:55.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[24814]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:32:55.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi017 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi017 to 5356M 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[24814]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[29581]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi017 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi017 to 5356M 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[29581]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:32:55.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:55 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:55 smithi129 ceph-mon[25657]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:32:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:55 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi017 2024-03-26T04:32:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:55 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi017 to 5356M 2024-03-26T04:32:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:55 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T04:32:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:55 smithi129 ceph-mon[25657]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:32:55.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:55 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:56.033 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:55 smithi017 sudo[39634]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:32:56.033 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:55 smithi017 sudo[39634]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:56.033 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:55 smithi017 sudo[39634]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:56.378 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39659]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:32:56.378 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39659]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:56.379 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39659]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:56.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39699]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:32:56.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39699]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:56.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39699]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:56.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[24814]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[29581]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:56.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:56 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:56 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:56 smithi129 ceph-mon[25657]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:32:56.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:56 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:32:56.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:56 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:32:56.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:56 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:32:56.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:56 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:56.975 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39743]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:32:56.975 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39743]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:56.975 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39743]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:57.240 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39765]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:32:57.241 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39765]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:57.241 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:56 smithi017 sudo[39765]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:57.497 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39768]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:32:57.497 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39768]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:57.498 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39768]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:57.763 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39774]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:32:57.763 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39774]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:57.763 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39774]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:58.030 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[29581]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[29581]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[29581]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[29581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39796]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39796]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 sudo[39796]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[24814]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[24814]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:58.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[24814]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:32:58.032 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:57 smithi017 ceph-mon[24814]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:32:58.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:57 smithi129 ceph-mon[25657]: pgmap v75: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:32:58.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:57 smithi129 ceph-mon[25657]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:32:58.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:57 smithi129 ceph-mon[25657]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T04:32:58.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:57 smithi129 ceph-mon[25657]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:32:58.293 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 sudo[39805]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:32:58.293 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 sudo[39805]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:58.294 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 sudo[39805]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:58.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 sudo[39817]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:32:58.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 sudo[39817]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:58.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 sudo[39817]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:58.788 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 sudo[27179]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:32:58.788 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 sudo[27179]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:58.788 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 sudo[27179]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:59.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 sudo[27182]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:32:59.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 sudo[27182]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T04:32:59.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 sudo[27182]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.147 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:58 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1273410203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4aa4945-f8dd-4b25-9d97-61be1ca2faf2"}]: dispatch 2024-03-26T04:32:59.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:32:59.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1273410203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4aa4945-f8dd-4b25-9d97-61be1ca2faf2"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:32:59.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:32:59.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:58 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1273410203' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4aa4945-f8dd-4b25-9d97-61be1ca2faf2"}]: dispatch 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1273410203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4aa4945-f8dd-4b25-9d97-61be1ca2faf2"}]': finished 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:00.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:32:59 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1590112338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:00.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:00.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:33:00.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1273410203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4aa4945-f8dd-4b25-9d97-61be1ca2faf2"}]': finished 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1590112338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:33:00.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:33:00.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1273410203' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4aa4945-f8dd-4b25-9d97-61be1ca2faf2"}]': finished 2024-03-26T04:33:00.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:33:00.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:00.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:00.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:32:59 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1590112338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:02.250 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:01 smithi017 ceph-mon[24814]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:02.250 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:01 smithi017 ceph-mon[29581]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:01 smithi129 ceph-mon[25657]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:04.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:03 smithi017 ceph-mon[29581]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:03 smithi017 ceph-mon[24814]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:03 smithi129 ceph-mon[25657]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:06.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:05 smithi017 ceph-mon[29581]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:06.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:05 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:33:06.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:05 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:06.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:05 smithi017 ceph-mon[24814]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:06.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:05 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:33:06.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:05 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:05 smithi129 ceph-mon[25657]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:06.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:05 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:33:06.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:05 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:07.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:06 smithi129 ceph-mon[25657]: Deploying daemon osd.3 on smithi017 2024-03-26T04:33:07.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:06 smithi017 ceph-mon[24814]: Deploying daemon osd.3 on smithi017 2024-03-26T04:33:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:06 smithi017 ceph-mon[29581]: Deploying daemon osd.3 on smithi017 2024-03-26T04:33:08.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:07 smithi017 ceph-mon[24814]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:08.167 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:07 smithi017 ceph-mon[29581]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:07 smithi129 ceph-mon[25657]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:10.031 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:09 smithi017 ceph-mon[24814]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:10.031 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:09 smithi017 ceph-mon[29581]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:10.358 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:09 smithi129 ceph-mon[25657]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:11.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[24814]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:11.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[29581]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:11.253 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:11 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:11 smithi129 ceph-mon[25657]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:11 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:11 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:11 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:11 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:13.663 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 3 on host 'smithi017' 2024-03-26T04:33:13.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:13.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:33:13.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:13.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:13.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:13.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:13.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:13 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:33:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:14.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:14.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:13 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:14.251 DEBUG:teuthology.orchestra.run.smithi017:osd.3> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T04:33:14.254 INFO:tasks.cephadm:Deploying osd.4 on smithi129 with /dev/vg_nvme/lv_4... 2024-03-26T04:33:14.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:33:14.875 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:14 smithi129 ceph-mon[25657]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:33:14.875 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:14 smithi129 ceph-mon[25657]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:33:14.875 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:14 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:14.875 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:14 smithi129 ceph-mon[25657]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:33:15.065 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[24814]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:33:15.065 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[24814]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:33:15.065 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.065 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[24814]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:33:15.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[29581]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:33:15.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[29581]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:33:15.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-mon[29581]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:33:15.066 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[41288]: 2024-03-26T04:33:14.803+0000 7f61a7966700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T04:33:15.066 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:33:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[41288]: 2024-03-26T04:33:14.808+0000 7f61a02f9700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:33:15.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[24814]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[24814]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[24814]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[29581]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[29581]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[29581]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T04:33:15.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:15 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:15 smithi129 ceph-mon[25657]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:15.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:15 smithi129 ceph-mon[25657]: from='osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-03-26T04:33:15.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:15 smithi129 ceph-mon[25657]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T04:33:15.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:15 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:15 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:15.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:15 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:33:16.298 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:33:16.299 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-26T04:33:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-26T04:33:16.299 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129815 s, 808 MB/s 2024-03-26T04:33:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-26T04:33:16.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-03-26T04:33:16.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:33:16.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:33:16.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[24814]: osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567] boot 2024-03-26T04:33:16.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[24814]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:33:17.000 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:17.000 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:33:17.000 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:33:17.000 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[29581]: osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567] boot 2024-03-26T04:33:17.000 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[29581]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:33:17.000 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:16 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:16 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:33:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:16 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:33:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:16 smithi129 ceph-mon[25657]: osd.3 [v2:172.21.15.17:6826/3671962567,v1:172.21.15.17:6827/3671962567] boot 2024-03-26T04:33:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:16 smithi129 ceph-mon[25657]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:33:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:16 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.092 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T04:33:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:18.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:18.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:19.129 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:18 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi017 2024-03-26T04:33:19.129 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:18 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi017 to 4017M 2024-03-26T04:33:19.130 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:18 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:33:19.130 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:18 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:33:19.130 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:18 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:19.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi017 2024-03-26T04:33:19.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi017 to 4017M 2024-03-26T04:33:19.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi017 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi017 to 4017M 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:33:19.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:18 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:20.034 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:19 smithi129 ceph-mon[25657]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:20.034 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:19 smithi129 ceph-mon[25657]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:33:20.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:19 smithi017 ceph-mon[29581]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:20.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:19 smithi017 ceph-mon[29581]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:33:20.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:19 smithi017 ceph-mon[24814]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:20.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:19 smithi017 ceph-mon[24814]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:33:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:20 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/1273970517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]: dispatch 2024-03-26T04:33:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:20 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]: dispatch 2024-03-26T04:33:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:20 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]': finished 2024-03-26T04:33:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:20 smithi129 ceph-mon[25657]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:33:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:20 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/1273970517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]: dispatch 2024-03-26T04:33:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]: dispatch 2024-03-26T04:33:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]': finished 2024-03-26T04:33:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[24814]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:33:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:21.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/1273970517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]: dispatch 2024-03-26T04:33:21.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]: dispatch 2024-03-26T04:33:21.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c36e2c8-de6d-48d4-b786-e710c5b729e7"}]': finished 2024-03-26T04:33:21.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[29581]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:33:21.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:20 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:21 smithi129 ceph-mon[25657]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/32897409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:21 smithi017 ceph-mon[29581]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/32897409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:21 smithi017 ceph-mon[24814]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/32897409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:24.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:23 smithi129 ceph-mon[25657]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:23 smithi017 ceph-mon[29581]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:24.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:23 smithi017 ceph-mon[24814]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:26.298 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:25 smithi129 ceph-mon[25657]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:25 smithi017 ceph-mon[29581]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:25 smithi017 ceph-mon[24814]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:27.235 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:33:27.235 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:33:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:33:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:28.141 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:27 smithi129 ceph-mon[25657]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:28.141 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:27 smithi129 ceph-mon[25657]: Deploying daemon osd.4 on smithi129 2024-03-26T04:33:28.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:27 smithi017 ceph-mon[29581]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:28.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:27 smithi017 ceph-mon[29581]: Deploying daemon osd.4 on smithi129 2024-03-26T04:33:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:27 smithi017 ceph-mon[24814]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:27 smithi017 ceph-mon[24814]: Deploying daemon osd.4 on smithi129 2024-03-26T04:33:30.281 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:29 smithi129 ceph-mon[25657]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:30.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:29 smithi017 ceph-mon[24814]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:30.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:29 smithi017 ceph-mon[29581]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:31 smithi129 ceph-mon[25657]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:31 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:31 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:31 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:31 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:32.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[24814]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:32.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[29581]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:32.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:31 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:34.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:33 smithi129 ceph-mon[25657]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:34.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:33 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:34.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:33 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:34.278 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 4 on host 'smithi129' 2024-03-26T04:33:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:33 smithi017 ceph-mon[24814]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:34.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:34.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:33 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:34.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:33 smithi017 ceph-mon[29581]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:34.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:34.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:33 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:35.067 DEBUG:teuthology.orchestra.run.smithi129:osd.4> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T04:33:35.070 INFO:tasks.cephadm:Deploying osd.5 on smithi129 with /dev/vg_nvme/lv_3... 2024-03-26T04:33:35.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: from='osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:33:35.321 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:35 smithi129 ceph-mon[25657]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:33:35.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: from='osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[24814]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: from='osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:33:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:35 smithi017 ceph-mon[29581]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:33:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-mon[25657]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:33:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-mon[25657]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:33:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:36.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-mon[25657]: from='osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:36.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-mon[25657]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:36.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[29111]: 2024-03-26T04:33:36.292+0000 7f80f7325700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T04:33:36.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:33:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[29111]: 2024-03-26T04:33:36.297+0000 7f80f34bf700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:33:36.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[29581]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:33:36.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[29581]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[29581]: from='osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[29581]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[24814]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[24814]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[24814]: from='osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:36.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:36 smithi017 ceph-mon[24814]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:37 smithi129 ceph-mon[25657]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:37 smithi129 ceph-mon[25657]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:33:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:37 smithi129 ceph-mon[25657]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T04:33:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:37 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:37.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:37 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:37.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[29581]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:37.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[29581]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[29581]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[24814]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[24814]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[24814]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:37.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:37 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:38.023 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:33:38.023 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:33:38.023 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-26T04:33:38.023 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-26T04:33:38.023 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120814 s, 868 MB/s 2024-03-26T04:33:38.023 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-26T04:33:38.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:38 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:33:38.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:38 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:33:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:38 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:38 smithi129 ceph-mon[25657]: osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081] boot 2024-03-26T04:33:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:38 smithi129 ceph-mon[25657]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:33:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:38 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:38.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[24814]: osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081] boot 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[24814]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:33:38.607 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:33:38.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:38.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[29581]: osd.4 [v2:172.21.15.129:6800/195292081,v1:172.21.15.129:6801/195292081] boot 2024-03-26T04:33:38.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[29581]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:33:38.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:38 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:33:39.533 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:39 smithi129 ceph-mon[25657]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:39.534 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:39 smithi129 ceph-mon[25657]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:33:39.534 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.534 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:39.534 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.534 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.622 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[29581]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[29581]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[24814]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[24814]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:39.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:40 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi129 2024-03-26T04:33:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:40 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-26T04:33:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:40 smithi129 ceph-mon[25657]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:33:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:40 smithi129 ceph-mon[25657]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:33:40.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi129 2024-03-26T04:33:40.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-26T04:33:40.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[24814]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:33:40.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[24814]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:33:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi129 2024-03-26T04:33:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi129 to 17088M 2024-03-26T04:33:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[29581]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:33:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:40 smithi017 ceph-mon[29581]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:33:41.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:41 smithi129 ceph-mon[25657]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-26T04:33:41.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:41 smithi129 ceph-mon[25657]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:33:41.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:41 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:33:41.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:41 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:33:41.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:41 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:41.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:41 smithi129 ceph-mon[25657]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[24814]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[24814]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[24814]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[29581]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-26T04:33:41.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[29581]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:33:41.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:33:41.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:33:41.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:41.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:41 smithi017 ceph-mon[29581]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T04:33:43.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:43 smithi129 ceph-mon[25657]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T04:33:43.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:43 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/2095550324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]: dispatch 2024-03-26T04:33:43.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:43 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]: dispatch 2024-03-26T04:33:43.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:43 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]': finished 2024-03-26T04:33:43.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:43 smithi129 ceph-mon[25657]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:33:43.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:33:43.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[24814]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/2095550324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]: dispatch 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]: dispatch 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]': finished 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[24814]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[29581]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/2095550324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]: dispatch 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]: dispatch 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6449fee9-f734-462f-ad3a-f8eb7df4aed9"}]': finished 2024-03-26T04:33:43.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[29581]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:33:43.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:33:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:44 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/2423219527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:44 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/2423219527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:44.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:44 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/2423219527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:33:46.244 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:45 smithi129 ceph-mon[25657]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-26T04:33:46.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:45 smithi017 ceph-mon[24814]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-26T04:33:46.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:45 smithi017 ceph-mon[29581]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-03-26T04:33:48.266 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:47 smithi129 ceph-mon[25657]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:33:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:47 smithi017 ceph-mon[24814]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:33:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:47 smithi017 ceph-mon[29581]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:33:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:33:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:33:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:33:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:48 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:33:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:33:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:33:49.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:49.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:33:49.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:33:49.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:33:49.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:48 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:49 smithi129 ceph-mon[25657]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:49 smithi129 ceph-mon[25657]: Deploying daemon osd.5 on smithi129 2024-03-26T04:33:50.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:49 smithi017 ceph-mon[24814]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:50.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:49 smithi017 ceph-mon[24814]: Deploying daemon osd.5 on smithi129 2024-03-26T04:33:50.274 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:49 smithi017 ceph-mon[29581]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:50.275 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:49 smithi017 ceph-mon[29581]: Deploying daemon osd.5 on smithi129 2024-03-26T04:33:52.213 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:51 smithi129 ceph-mon[25657]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:51 smithi017 ceph-mon[24814]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:51 smithi017 ceph-mon[29581]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:54.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:53 smithi129 ceph-mon[25657]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:54.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:53 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:54.314 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:53 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:54.314 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:53 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:54.314 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:53 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[24814]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[29581]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:53 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:55 smithi129 ceph-mon[25657]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:55 smithi017 ceph-mon[24814]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:56.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:55 smithi017 ceph-mon[29581]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:57.186 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 5 on host 'smithi129' 2024-03-26T04:33:57.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:57.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:33:57.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:57.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:57.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:57 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:57.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:57.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:57 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:57.846 DEBUG:teuthology.orchestra.run.smithi129:osd.5> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T04:33:57.849 INFO:tasks.cephadm:Deploying osd.6 on smithi129 with /dev/vg_nvme/lv_2... 2024-03-26T04:33:57.849 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:33:59.101 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-mon[25657]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:33:59.101 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-mon[25657]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:33:59.101 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:33:59.101 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-mon[25657]: from='osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:59.101 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-mon[25657]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:59.101 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[31996]: 2024-03-26T04:33:59.004+0000 7fd2861fc700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T04:33:59.102 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:33:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[31996]: 2024-03-26T04:33:59.010+0000 7fd282b97700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:33:59.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[29581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:33:59.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[29581]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:33:59.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[29581]: from='osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[29581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[24814]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[24814]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[24814]: from='osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:33:59.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:33:59 smithi017 ceph-mon[24814]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:00.208 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:34:00.208 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:00.208 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T04:34:00.208 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.208 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.209 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453] boot 2024-03-26T04:34:00.209 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:34:00.209 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:00 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453] boot 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T04:34:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: osd.5 [v2:172.21.15.129:6808/1378709453,v1:172.21.15.129:6809/1378709453] boot 2024-03-26T04:34:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:34:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:00 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:01 smithi129 ceph-mon[25657]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:34:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:34:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:34:01.395 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-26T04:34:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-26T04:34:01.395 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123162 s, 851 MB/s 2024-03-26T04:34:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-26T04:34:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:34:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:34:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[24814]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:01.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:01 smithi017 ceph-mon[29581]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:34:02.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-03-26T04:34:02.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:02 smithi129 ceph-mon[25657]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:02.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:02 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi129 2024-03-26T04:34:02.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:02 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi129 to 8544M 2024-03-26T04:34:02.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:02 smithi129 ceph-mon[25657]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:34:02.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[24814]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:02.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi129 2024-03-26T04:34:02.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi129 to 8544M 2024-03-26T04:34:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[24814]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:34:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[29581]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi129 2024-03-26T04:34:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi129 to 8544M 2024-03-26T04:34:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:02 smithi017 ceph-mon[29581]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:34:03.294 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:03 smithi129 ceph-mon[25657]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:03.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:03 smithi017 ceph-mon[24814]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:03.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:03 smithi017 ceph-mon[29581]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:04.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:04 smithi129 ceph-mon[25657]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:34:04.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:04 smithi129 ceph-mon[25657]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:04.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:04 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:34:04.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:04 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:34:04.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:04 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[24814]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:34:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[24814]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[29581]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[29581]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:34:04.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:04 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:05.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:05 smithi129 ceph-mon[25657]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:05.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:05 smithi129 ceph-mon[25657]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T04:34:05.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:05 smithi017 ceph-mon[24814]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:05.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:05 smithi017 ceph-mon[24814]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T04:34:05.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:05 smithi017 ceph-mon[29581]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:05.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:05 smithi017 ceph-mon[29581]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/2576758079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]: dispatch 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]: dispatch 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]': finished 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:07.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:07 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/3596549326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:34:07.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:07.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/2576758079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]': finished 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/3596549326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/2576758079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6badc639-2f90-40cf-b37e-fd17dbb0fca6"}]': finished 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:07.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:07 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/3596549326' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:34:09.377 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:09 smithi129 ceph-mon[25657]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:09.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:09 smithi017 ceph-mon[29581]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:09.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:09 smithi017 ceph-mon[24814]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:12.107 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:11 smithi129 ceph-mon[25657]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:12.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:11 smithi017 ceph-mon[24814]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:12.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:11 smithi017 ceph-mon[29581]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:13.102 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:12 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:34:13.103 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:12 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:12 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:34:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:12 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:13.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:12 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:34:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:12 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:14.286 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:13 smithi129 ceph-mon[25657]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:14.286 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:13 smithi129 ceph-mon[25657]: Deploying daemon osd.6 on smithi129 2024-03-26T04:34:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:13 smithi017 ceph-mon[29581]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:13 smithi017 ceph-mon[29581]: Deploying daemon osd.6 on smithi129 2024-03-26T04:34:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:13 smithi017 ceph-mon[24814]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:14.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:13 smithi017 ceph-mon[24814]: Deploying daemon osd.6 on smithi129 2024-03-26T04:34:16.386 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:15 smithi129 ceph-mon[25657]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:16.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:15 smithi017 ceph-mon[24814]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:16.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:15 smithi017 ceph-mon[29581]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:18.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:17 smithi129 ceph-mon[25657]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:18.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:18.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:18.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:18.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:17 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[24814]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[29581]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:18.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:17 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:19.827 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 6 on host 'smithi129' 2024-03-26T04:34:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:19 smithi129 ceph-mon[25657]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:19 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:20.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:19 smithi017 ceph-mon[24814]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:20.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:19 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:20.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:19 smithi017 ceph-mon[29581]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:20.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:19 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:20.732 DEBUG:teuthology.orchestra.run.smithi129:osd.6> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T04:34:20.735 INFO:tasks.cephadm:Deploying osd.7 on smithi129 with /dev/vg_nvme/lv_1... 2024-03-26T04:34:20.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:34:21.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:21 smithi129 ceph-mon[25657]: from='osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:34:21.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:21 smithi129 ceph-mon[25657]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:34:21.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:21 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:21 smithi017 ceph-mon[24814]: from='osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:34:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:21 smithi017 ceph-mon[24814]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:34:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:21 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:21 smithi017 ceph-mon[29581]: from='osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:34:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:21 smithi017 ceph-mon[29581]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:34:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:21 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-mon[25657]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-mon[25657]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:34:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-mon[25657]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:34:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-mon[25657]: from='osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-mon[25657]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:22.336 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[34880]: 2024-03-26T04:34:22.015+0000 7fdf97169700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T04:34:22.336 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:34:22 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[34880]: 2024-03-26T04:34:22.022+0000 7fdf91b00700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:34:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[24814]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[24814]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:34:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[24814]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[24814]: from='osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[24814]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[29581]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[29581]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[29581]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[29581]: from='osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:22 smithi017 ceph-mon[29581]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:22.704 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:34:22.704 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:34:22.704 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-26T04:34:22.704 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-26T04:34:22.704 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124812 s, 840 MB/s 2024-03-26T04:34:22.704 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-26T04:34:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T04:34:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989] boot 2024-03-26T04:34:23.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:34:23.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:23 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:23.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T04:34:23.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989] boot 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: osd.6 [v2:172.21.15.129:6816/1036895989,v1:172.21.15.129:6817/1036895989] boot 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:34:23.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:23 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:34:23.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-03-26T04:34:24.266 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:24 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:34:24.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:24 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:34:24.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:24 smithi129 ceph-mon[25657]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:24.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:24 smithi129 ceph-mon[25657]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:34:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:34:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:34:24.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[24814]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:24.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[24814]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:34:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:34:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:34:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[29581]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:34:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:24 smithi017 ceph-mon[29581]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.278 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.279 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:26.279 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:26.279 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:26.279 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:26 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:26.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:26 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:27.116 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:27 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi129 2024-03-26T04:34:27.117 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:27 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi129 to 5696M 2024-03-26T04:34:27.117 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:27 smithi129 ceph-mon[25657]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:34:27.117 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:27 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:34:27.117 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:27 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:34:27.117 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:27 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi129 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi129 to 5696M 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[24814]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi129 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi129 to 5696M 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[29581]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:34:27.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:34:27.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:27 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:28.158 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:28 smithi129 ceph-mon[25657]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:28.158 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:28 smithi129 ceph-mon[25657]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:28.159 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:28 smithi129 ceph-mon[25657]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T04:34:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:28 smithi017 ceph-mon[24814]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:28 smithi017 ceph-mon[24814]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:28 smithi017 ceph-mon[24814]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T04:34:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:28 smithi017 ceph-mon[29581]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:28 smithi017 ceph-mon[29581]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:28 smithi017 ceph-mon[29581]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T04:34:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:29 smithi129 ceph-mon[25657]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:29 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/1647217578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]: dispatch 2024-03-26T04:34:29.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:29 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]: dispatch 2024-03-26T04:34:29.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:29 smithi129 ceph-mon[25657]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]': finished 2024-03-26T04:34:29.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:29 smithi129 ceph-mon[25657]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:34:29.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:29 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:29.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[24814]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:29.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/1647217578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]: dispatch 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]: dispatch 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[24814]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]': finished 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[24814]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[29581]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/1647217578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]: dispatch 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]: dispatch 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[29581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9c73505-35f9-41f0-a439-65dbb7122d6a"}]': finished 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[29581]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:34:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:29 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:30 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/2489493417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:34:30.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:30 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/2489493417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:34:30.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:30 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/2489493417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:34:32.226 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:31 smithi129 ceph-mon[25657]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-26T04:34:32.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:31 smithi017 ceph-mon[24814]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-26T04:34:32.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:31 smithi017 ceph-mon[29581]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-26T04:34:34.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:33 smithi129 ceph-mon[25657]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:34:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:33 smithi017 ceph-mon[24814]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:34:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:33 smithi017 ceph-mon[29581]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:34:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:34:35.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:34 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:35.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:34 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:34:35.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:34 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:35.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:34 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:34:35.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:34 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:36.294 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:35 smithi129 ceph-mon[25657]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T04:34:36.295 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:35 smithi129 ceph-mon[25657]: Deploying daemon osd.7 on smithi129 2024-03-26T04:34:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:35 smithi017 ceph-mon[24814]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T04:34:36.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:35 smithi017 ceph-mon[24814]: Deploying daemon osd.7 on smithi129 2024-03-26T04:34:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:35 smithi017 ceph-mon[29581]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T04:34:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:35 smithi017 ceph-mon[29581]: Deploying daemon osd.7 on smithi129 2024-03-26T04:34:38.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:37 smithi129 ceph-mon[25657]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-26T04:34:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:37 smithi017 ceph-mon[24814]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-26T04:34:38.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:37 smithi017 ceph-mon[29581]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[24814]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[29581]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T04:34:40.258 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:40.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:40.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:40.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:39 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:40.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:39 smithi129 ceph-mon[25657]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T04:34:40.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:40.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:40.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:40.267 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:39 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:42.337 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:42 smithi129 ceph-mon[25657]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:42.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:42 smithi017 ceph-mon[24814]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:42.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:42 smithi017 ceph-mon[29581]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:42.969 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 7 on host 'smithi129' 2024-03-26T04:34:43.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:43.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:43 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:43.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:43.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:34:43.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:34:43.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:43.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:43.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:43.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:43 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:43.631 DEBUG:teuthology.orchestra.run.smithi129:osd.7> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T04:34:43.634 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T04:34:43.634 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:34:44.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[24814]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[24814]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[24814]: from='osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[24814]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[29581]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[29581]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[29581]: from='osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:44 smithi017 ceph-mon[29581]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:44.540 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-mon[25657]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:34:44.540 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-mon[25657]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:34:44.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:44.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-mon[25657]: from='osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:44.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-mon[25657]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:34:44.541 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[37864]: 2024-03-26T04:34:44.198+0000 7f3fbd649700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T04:34:44.541 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:34:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[37864]: 2024-03-26T04:34:44.205+0000 7f3fb7fe0700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653] boot 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:34:45.258 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653] boot 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:34:45.259 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:45 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: osdmap e54: 8 total, 7 up, 8 in 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: osd.7 [v2:172.21.15.129:6824/3889091653,v1:172.21.15.129:6825/3889091653] boot 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:34:45.615 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:45 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:34:45.656 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: purged_snaps scrub starts 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: purged_snaps scrub ok 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2730049303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: purged_snaps scrub starts 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: purged_snaps scrub ok 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2730049303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.255 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:46 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.276 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711427684,"num_in_osds":8,"osd_in_since":1711427668,"num_remapped_pgs":1} 2024-03-26T04:34:46.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:34:46.499 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: purged_snaps scrub starts 2024-03-26T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: purged_snaps scrub ok 2024-03-26T04:34:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2730049303' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:46 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:48.178 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:34:48.178 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":58,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","created":"2024-03-26T04:29:59.396661+0000","modified":"2024-03-26T04:34:47.993082+0000","last_up_change":"2024-03-26T04:34:44.985487+0000","last_in_change":"2024-03-26T04:34:28.871858+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:32:54.301748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"55282a5c-04c7-4f56-ae85-1f7b13034dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6803","nonce":857707498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6805","nonce":857707498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6809","nonce":857707498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6807","nonce":857707498}]},"public_addr":"172.21.15.17:6803/857707498","cluster_addr":"172.21.15.17:6805/857707498","heartbeat_back_addr":"172.21.15.17:6809/857707498","heartbeat_front_addr":"172.21.15.17:6807/857707498","state":["exists","up"]},{"osd":1,"uuid":"8b8f13d8-440b-4859-adac-d5dba48d4677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6811","nonce":3878069074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6813","nonce":3878069074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6817","nonce":3878069074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6815","nonce":3878069074}]},"public_addr":"172.21.15.17:6811/3878069074","cluster_addr":"172.21.15.17:6813/3878069074","heartbeat_back_addr":"172.21.15.17:6817/3878069074","heartbeat_front_addr":"172.21.15.17:6815/3878069074","state":["exists","up"]},{"osd":2,"uuid":"bbff43d2-2aec-401f-897d-afc6b7ab595b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6819","nonce":24179605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6821","nonce":24179605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6825","nonce":24179605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6823","nonce":24179605}]},"public_addr":"172.21.15.17:6819/24179605","cluster_addr":"172.21.15.17:6821/24179605","heartbeat_back_addr":"172.21.15.17:6825/24179605","heartbeat_front_addr":"172.21.15.17:6823/24179605","state":["exists","up"]},{"osd":3,"uuid":"c4aa4945-f8dd-4b25-9d97-61be1ca2faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6827","nonce":3671962567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6829","nonce":3671962567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6833","nonce":3671962567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6831","nonce":3671962567}]},"public_addr":"172.21.15.17:6827/3671962567","cluster_addr":"172.21.15.17:6829/3671962567","heartbeat_back_addr":"172.21.15.17:6833/3671962567","heartbeat_front_addr":"172.21.15.17:6831/3671962567","state":["exists","up"]},{"osd":4,"uuid":"6c36e2c8-de6d-48d4-b786-e710c5b729e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6801","nonce":195292081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6803","nonce":195292081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6807","nonce":195292081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6805","nonce":195292081}]},"public_addr":"172.21.15.129:6801/195292081","cluster_addr":"172.21.15.129:6803/195292081","heartbeat_back_addr":"172.21.15.129:6807/195292081","heartbeat_front_addr":"172.21.15.129:6805/195292081","state":["exists","up"]},{"osd":5,"uuid":"6449fee9-f734-462f-ad3a-f8eb7df4aed9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6809","nonce":1378709453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6811","nonce":1378709453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6815","nonce":1378709453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6813","nonce":1378709453}]},"public_addr":"172.21.15.129:6809/1378709453","cluster_addr":"172.21.15.129:6811/1378709453","heartbeat_back_addr":"172.21.15.129:6815/1378709453","heartbeat_front_addr":"172.21.15.129:6813/1378709453","state":["exists","up"]},{"osd":6,"uuid":"6badc639-2f90-40cf-b37e-fd17dbb0fca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6817","nonce":1036895989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6819","nonce":1036895989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6823","nonce":1036895989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6821","nonce":1036895989}]},"public_addr":"172.21.15.129:6817/1036895989","cluster_addr":"172.21.15.129:6819/1036895989","heartbeat_back_addr":"172.21.15.129:6823/1036895989","heartbeat_front_addr":"172.21.15.129:6821/1036895989","state":["exists","up"]},{"osd":7,"uuid":"d9c73505-35f9-41f0-a439-65dbb7122d6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6825","nonce":3889091653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6827","nonce":3889091653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6831","nonce":3889091653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6829","nonce":3889091653}]},"public_addr":"172.21.15.129:6825/3889091653","cluster_addr":"172.21.15.129:6827/3889091653","heartbeat_back_addr":"172.21.15.129:6831/3889091653","heartbeat_front_addr":"172.21.15.129:6829/3889091653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:04.414875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:27.268615+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:50.557826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:14.241162+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:35.508372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:58.049236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:34:21.309267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:34:43.727882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:6801/4126564882":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6800/4199384931":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/4101203109":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6801/4199384931":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/2717172247":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6800/4126564882":"2024-03-27T04:30:47.925915+0000","172.21.15.17:0/27415178":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1886501174":"2024-03-27T04:30:47.925915+0000","172.21.15.17:0/3729939084":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1619556253":"2024-03-27T04:30:25.908881+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:34:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:47 smithi129 ceph-mon[25657]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:47 smithi129 ceph-mon[25657]: Detected new or changed devices on smithi129 2024-03-26T04:34:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:47 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi129 to 4272M 2024-03-26T04:34:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:47 smithi129 ceph-mon[25657]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:34:48.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:48.004+0000 7f1673a1d700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[24814]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[24814]: Detected new or changed devices on smithi129 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi129 to 4272M 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[24814]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[29581]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[29581]: Detected new or changed devices on smithi129 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi129 to 4272M 2024-03-26T04:34:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:47 smithi017 ceph-mon[29581]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:34:48.787 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:32:54.301748+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-26T04:34:48.787 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T04:34:49.013 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[29581]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3399775745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[24814]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:34:49.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:49 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3399775745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:34:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:49 smithi129 ceph-mon[25657]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:34:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:49 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:34:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:49 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:34:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:49 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3399775745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:34:50.339 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:50 smithi017 ceph-mon[29581]: pgmap v168: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-03-26T04:34:50.340 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:50 smithi017 ceph-mon[29581]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:34:50.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:50 smithi017 ceph-mon[24814]: pgmap v168: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-03-26T04:34:50.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:50 smithi017 ceph-mon[24814]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:34:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:50 smithi129 ceph-mon[25657]: pgmap v168: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-03-26T04:34:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:50 smithi129 ceph-mon[25657]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:34:50.758 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2024-03-26T04:34:51.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:51 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1096863273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:34:51.314 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:51 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1096863273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:34:51.341 INFO:tasks.cephadm:Adding prometheus.a on smithi129 2024-03-26T04:34:51.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi129=a' 2024-03-26T04:34:51.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:51 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1096863273' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:34:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:52 smithi129 ceph-mon[25657]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-26T04:34:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:52 smithi017 ceph-mon[24814]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-26T04:34:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:52 smithi017 ceph-mon[29581]: pgmap v170: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-26T04:34:53.279 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled prometheus update... 2024-03-26T04:34:53.826 DEBUG:teuthology.orchestra.run.smithi129:prometheus.a> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T04:34:53.868 INFO:tasks.cephadm:Adding node-exporter.a on smithi017 2024-03-26T04:34:53.868 INFO:tasks.cephadm:Adding node-exporter.b on smithi129 2024-03-26T04:34:53.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi017=a;smithi129=b' 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:34:54.356 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:34:54.356 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:54.356 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:54.356 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:54.356 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' 2024-03-26T04:34:54.357 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:34:54.472 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ignoring --setuser ceph since I am not root 2024-03-26T04:34:54.472 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ignoring --setgroup ceph since I am not root 2024-03-26T04:34:54.473 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-mgr[26856]: -- 172.21.15.129:0/3911098349 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be3c42e340 con 0x55be3c406c00 2024-03-26T04:34:54.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setuser ceph since I am not root 2024-03-26T04:34:54.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setgroup ceph since I am not root 2024-03-26T04:34:54.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-mgr[25046]: -- 172.21.15.17:0/3068084571 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5634d93c2340 con 0x5634d939a800 2024-03-26T04:34:55.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:54.825+0000 7f34a332f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:34:55.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:54.918+0000 7f34a332f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:34:55.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-mon[25657]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:55.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-mon[25657]: Saving service prometheus spec with placement smithi129=a;count:1 2024-03-26T04:34:55.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-mon[25657]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:34:55.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-mon[25657]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:34:55.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[29581]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:55.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[29581]: Saving service prometheus spec with placement smithi129=a;count:1 2024-03-26T04:34:55.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[29581]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:34:55.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[29581]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:34:55.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[24814]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:34:55.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[24814]: Saving service prometheus spec with placement smithi129=a;count:1 2024-03-26T04:34:55.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[24814]: from='mgr.14152 172.21.15.17:0/1937511448' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:34:55.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-mon[24814]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:34:55.207 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:54.843+0000 7f126e520000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:34:55.207 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:54.935+0000 7f126e520000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:34:55.862 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:55.525+0000 7f34a332f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:34:55.862 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:55.618+0000 7f34a332f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:34:55.862 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:55.708+0000 7f34a332f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:34:55.862 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:55.861+0000 7f34a332f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:34:55.869 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:55.541+0000 7f126e520000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:34:55.869 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:55.632+0000 7f126e520000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:34:55.869 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:55.716+0000 7f126e520000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:34:55.869 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:55.868+0000 7f126e520000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:34:56.134 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:55.941+0000 7f34a332f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:34:56.134 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:56.047+0000 7f34a332f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:34:56.146 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:55 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:55.950+0000 7f126e520000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:34:56.146 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:56.061+0000 7f126e520000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:34:56.146 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:56.145+0000 7f126e520000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:34:56.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:56.130+0000 7f34a332f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:34:56.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:56.293+0000 7f34a332f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:34:56.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:56.377+0000 7f34a332f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:34:56.398 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:56.311+0000 7f126e520000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:34:56.398 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:56.397+0000 7f126e520000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:34:57.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:57 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:57.445+0000 7f126e520000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:34:57.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:57 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:57.583+0000 7f126e520000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:34:57.711 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:57.396+0000 7f34a332f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:34:57.712 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:57.524+0000 7f34a332f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:34:58.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:57.711+0000 7f34a332f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:34:58.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:57.799+0000 7f34a332f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:34:58.146 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:57.933+0000 7f34a332f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:34:58.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:57 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:57.779+0000 7f126e520000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:34:58.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:57 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:57.868+0000 7f126e520000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:34:58.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:58.008+0000 7f126e520000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:34:58.554 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:58.251+0000 7f34a332f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:34:58.554 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:58.469+0000 7f34a332f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:34:58.646 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:58.331+0000 7f126e520000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:34:58.646 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:58.558+0000 7f126e520000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:34:58.646 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:58.645+0000 7f126e520000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:34:58.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:58.553+0000 7f34a332f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:34:58.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:58.720+0000 7f34a332f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:34:58.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:58.821+0000 7f126e520000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:34:59.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:59.633+0000 7f34a332f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:35:00.069 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:34:59 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:34:59.763+0000 7f126e520000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:35:00.070 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:35:00.069+0000 7f126e520000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:35:00.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:34:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:34:59.925+0000 7f34a332f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:35:00.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:35:00.019+0000 7f34a332f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:35:00.185 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:35:00.169+0000 7f34a332f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:35:00.185 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: [26/Mar/2024:04:35:00] ENGINE Bus STARTING 2024-03-26T04:35:00.322 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:35:00.167+0000 7f126e520000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:35:00.323 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:35:00.322+0000 7f126e520000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:35:00.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: CherryPy Checker: 2024-03-26T04:35:00.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: The Application mounted at '' has an empty config. 2024-03-26T04:35:00.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: 2024-03-26T04:35:00.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: [26/Mar/2024:04:35:00] ENGINE Serving on http://:::9283 2024-03-26T04:35:00.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: [26/Mar/2024:04:35:00] ENGINE Bus STARTED 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: Standby manager daemon x restarted 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: Standby manager daemon x started 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: Active manager daemon y restarted 2024-03-26T04:35:01.335 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: Activating manager daemon y 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[24814]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:35:01] ENGINE Bus STARTING 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: CherryPy Checker: 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: The Application mounted at '' has an empty config. 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:35:01] ENGINE Serving on http://:::9283 2024-03-26T04:35:01.336 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: Standby manager daemon x restarted 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: Standby manager daemon x started 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: Active manager daemon y restarted 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: Activating manager daemon y 2024-03-26T04:35:01.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:00 smithi017 ceph-mon[29581]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:35:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: Standby manager daemon x restarted 2024-03-26T04:35:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: Standby manager daemon x started 2024-03-26T04:35:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:35:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:35:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:35:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/140324421' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:35:01.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: Active manager daemon y restarted 2024-03-26T04:35:01.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: Activating manager daemon y 2024-03-26T04:35:01.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:00 smithi129 ceph-mon[25657]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:35:01.590 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:35:01] ENGINE Bus STARTED 2024-03-26T04:35:02.018 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled node-exporter update... 2024-03-26T04:35:02.198 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:35:01] ENGINE Bus STARTING 2024-03-26T04:35:02.198 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:02 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:35:02] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:35:02.198 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:02 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:35:02] ENGINE Bus STARTED 2024-03-26T04:35:02.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: mgrmap e17: y(active, starting, since 0.664896s), standbys: x 2024-03-26T04:35:02.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:35:02.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: Manager daemon y is now available 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: mgrmap e17: y(active, starting, since 0.664896s), standbys: x 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:35:02.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:35:02.201 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: Manager daemon y is now available 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:35:02.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:35:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: mgrmap e17: y(active, starting, since 0.664896s), standbys: x 2024-03-26T04:35:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:35:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:35:02.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: Manager daemon y is now available 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:35:02.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:35:03.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: [26/Mar/2024:04:35:01] ENGINE Bus STARTING 2024-03-26T04:35:03.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: mgrmap e18: y(active, since 1.67223s), standbys: x 2024-03-26T04:35:03.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi017=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:03.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:03.078 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: Saving service node-exporter spec with placement smithi017=a;smithi129=b;count:2 2024-03-26T04:35:03.079 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.079 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: [26/Mar/2024:04:35:02] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:35:03.079 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: [26/Mar/2024:04:35:02] ENGINE Bus STARTED 2024-03-26T04:35:03.079 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.079 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.104 DEBUG:teuthology.orchestra.run.smithi017:node-exporter.a> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: [26/Mar/2024:04:35:01] ENGINE Bus STARTING 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: mgrmap e18: y(active, since 1.67223s), standbys: x 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi017=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: Saving service node-exporter spec with placement smithi017=a;smithi129=b;count:2 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.105 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: [26/Mar/2024:04:35:02] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: [26/Mar/2024:04:35:02] ENGINE Bus STARTED 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: [26/Mar/2024:04:35:01] ENGINE Bus STARTING 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: mgrmap e18: y(active, since 1.67223s), standbys: x 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi017=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: Saving service node-exporter spec with placement smithi017=a;smithi129=b;count:2 2024-03-26T04:35:03.106 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: [26/Mar/2024:04:35:02] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:35:03.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: [26/Mar/2024:04:35:02] ENGINE Bus STARTED 2024-03-26T04:35:03.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:03.108 DEBUG:teuthology.orchestra.run.smithi129:node-exporter.b> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T04:35:03.110 INFO:tasks.cephadm:Adding alertmanager.a on smithi017 2024-03-26T04:35:03.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi017=a' 2024-03-26T04:35:04.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:35:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:04 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.261 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:04.261 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-26T04:35:04.261 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.261 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.261 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:35:04.262 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:04.262 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:04 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:05 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:35:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:05 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:35:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:05 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:35:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:05 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:05 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.190 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled alertmanager update... 2024-03-26T04:35:05.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:35:05.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:35:05.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:05 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:05.843 DEBUG:teuthology.orchestra.run.smithi017:alertmanager.a> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T04:35:05.846 INFO:tasks.cephadm:Adding grafana.a on smithi129 2024-03-26T04:35:05.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi129=a' 2024-03-26T04:35:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[24814]: Deploying daemon prometheus.a on smithi129 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[24814]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[24814]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[24814]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[29581]: Deploying daemon prometheus.a on smithi129 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[29581]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[29581]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[29581]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-03-26T04:35:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:06 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:06.550 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:06 smithi129 ceph-mon[25657]: Deploying daemon prometheus.a on smithi129 2024-03-26T04:35:06.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:06 smithi129 ceph-mon[25657]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:06.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:06 smithi129 ceph-mon[25657]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:06.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:06 smithi129 ceph-mon[25657]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-03-26T04:35:06.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:06 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:08 smithi017 ceph-mon[24814]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:08.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:08 smithi017 ceph-mon[29581]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:08.507 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:08 smithi129 ceph-mon[25657]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:09.202 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled grafana update... 2024-03-26T04:35:09.980 DEBUG:teuthology.orchestra.run.smithi129:grafana.a> sudo journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T04:35:09.982 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T04:35:09.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:35:10.267 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[29581]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[29581]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[29581]: Saving service grafana spec with placement smithi129=a;count:1 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[24814]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[24814]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[24814]: Saving service grafana spec with placement smithi129=a;count:1 2024-03-26T04:35:10.268 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:10 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:10 smithi129 ceph-mon[25657]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:10 smithi129 ceph-mon[25657]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:35:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:10 smithi129 ceph-mon[25657]: Saving service grafana spec with placement smithi129=a;count:1 2024-03-26T04:35:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:10 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:12.047 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-03-26T04:35:12.047 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCAUAJmEMWqAhAADIl+721nNq8l/WO1y2Mgcw== 2024-03-26T04:35:12.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[29581]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:12.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3299006990' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:12.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:12.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:35:12.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[24814]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:12.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3299006990' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:12.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:12.343 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:12 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:35:12.616 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-03-26T04:35:12.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T04:35:12.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T04:35:12.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:12 smithi129 ceph-mon[25657]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:12.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:12 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3299006990' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:12.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:12 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:12.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:12 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:35:12.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:35:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:14 smithi129 ceph-mon[25657]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:14.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:14 smithi017 ceph-mon[24814]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:14.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:14 smithi017 ceph-mon[29581]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:16.340 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:15 smithi129 ceph-mon[25657]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:16.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:15 smithi017 ceph-mon[24814]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:16.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:15 smithi017 ceph-mon[29581]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:16.626 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:16 smithi129 podman[40710]: 2024-03-26 04:35:16.475113301 +0000 UTC m=+0.022523412 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T04:35:16.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:16 smithi129 podman[40710]: 2024-03-26 04:35:16.709172221 +0000 UTC m=+0.256582314 container create 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:35:16.987 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2024-03-26T04:35:16.988 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQCEUAJmzeGUOhAAtpVI+Q4YgqeQzDSJQat2tg== 2024-03-26T04:35:17.372 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:16 smithi129 ceph-mon[25657]: from='client.? 172.21.15.129:0/461669709' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:17.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:16 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:17.373 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:16 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:35:17.373 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:16 smithi129 podman[40710]: 2024-03-26 04:35:16.995269783 +0000 UTC m=+0.542679879 container init 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:35:17.373 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:16 smithi129 podman[40710]: 2024-03-26 04:35:16.998267901 +0000 UTC m=+0.545678003 container start 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:35:17.373 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.026Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.026Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.026Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.026Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi129 (none))" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.027Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.027Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.028Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.029Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.029Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.032Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.032Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.893µs 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.032Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.032Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T04:35:17.374 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.032Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.916µs wal_replay_duration=137.521µs total_replay_duration=175.689µs 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.033Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.033Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.033Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.047Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.337855ms db_storage=611ns remote_storage=1.265µs web_handler=210ns query_engine=434ns scrape=441.161µs scrape_sd=15.589µs notify=930ns notify_sd=1.379µs rules=13.718604ms 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:35:17.047Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 bash[40710]: 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 2024-03-26T04:35:17.375 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:35:17 smithi129 systemd[1]: Started Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:35:17.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:16 smithi017 ceph-mon[24814]: from='client.? 172.21.15.129:0/461669709' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:17.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:16 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:17.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:16 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:35:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:16 smithi017 ceph-mon[29581]: from='client.? 172.21.15.129:0/461669709' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:16 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:35:17.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:16 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:35:17.665 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-26T04:35:17.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T04:35:17.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T04:35:17.710 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T04:35:17.710 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T04:35:17.710 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T04:35:18.010 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:18.026 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:17 smithi017 systemd[1]: Starting Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:35:18.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:18 smithi017 ceph-mon[24814]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:18.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:18 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:18.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:18 smithi017 ceph-mon[24814]: Deploying daemon node-exporter.a on smithi017 2024-03-26T04:35:18.364 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:18 smithi017 ceph-mon[29581]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:18.364 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:18 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:18.364 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:18 smithi017 ceph-mon[29581]: Deploying daemon node-exporter.a on smithi017 2024-03-26T04:35:18.365 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:18 smithi017 bash[44212]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-26T04:35:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:18 smithi129 ceph-mon[25657]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:18 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:18 smithi129 ceph-mon[25657]: Deploying daemon node-exporter.a on smithi017 2024-03-26T04:35:18.705 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:18 smithi017 bash[44212]: Getting image source signatures 2024-03-26T04:35:18.706 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:18 smithi017 bash[44212]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-26T04:35:18.706 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:18 smithi017 bash[44212]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-26T04:35:18.706 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:18 smithi017 bash[44212]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-26T04:35:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:20 smithi129 ceph-mon[25657]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:20 smithi017 ceph-mon[24814]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:20.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:20 smithi017 ceph-mon[29581]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:20.456 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:20 smithi017 bash[44212]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-26T04:35:20.869 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:20 smithi017 bash[44212]: Writing manifest to image destination 2024-03-26T04:35:21.066 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:35:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3742134418' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:35:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:21 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:21.438 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3742134418' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:35:21.438 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:21.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3742134418' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:35:21.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:21.439 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 podman[44212]: 2024-03-26 04:35:21.041156433 +0000 UTC m=+3.125959688 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T04:35:21.439 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 podman[44212]: 2024-03-26 04:35:21.180598758 +0000 UTC m=+3.265401977 container create af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:35:21.862 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":19,"active_gid":14448,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":1340511436},{"type":"v1","addr":"172.21.15.17:6801","nonce":1340511436}]},"active_addr":"172.21.15.17:6801/1340511436","active_change":"2024-03-26T04:35:00.336583+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24335,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.17:8443/","prometheus":"http://172.21.15.17:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3616639726}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3507232506}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":912684274}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3847572043}]}]}} 2024-03-26T04:35:21.865 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T04:35:21.865 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T04:35:21.866 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:35:22.098 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 podman[44212]: 2024-03-26 04:35:21.954103581 +0000 UTC m=+4.038906836 container init af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:35:22.098 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 podman[44212]: 2024-03-26 04:35:21.957248119 +0000 UTC m=+4.042051341 container start af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.961Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.961Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.961Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.961Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T04:35:22.099 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T04:35:22.100 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T04:35:22.100 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T04:35:22.100 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T04:35:22.100 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T04:35:22.100 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T04:35:22.101 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T04:35:22.102 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T04:35:22.102 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T04:35:22.102 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T04:35:22.102 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T04:35:22.102 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T04:35:22.102 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T04:35:22.103 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T04:35:22.103 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T04:35:22.103 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T04:35:22.103 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:21 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[44387]: ts=2024-03-26T04:35:21.962Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:22 smithi017 bash[44212]: af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 2024-03-26T04:35:22.104 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:35:22 smithi017 systemd[1]: Started Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:35:22.104 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:22 smithi129 ceph-mon[25657]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:22 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:22 smithi017 ceph-mon[24814]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:22 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:22 smithi017 ceph-mon[29581]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:22 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:23.144 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:22 smithi129 bash[41010]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-26T04:35:23.346 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:23 smithi017 ceph-mon[24814]: Deploying daemon node-exporter.b on smithi129 2024-03-26T04:35:23.347 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:23 smithi017 ceph-mon[29581]: Deploying daemon node-exporter.b on smithi129 2024-03-26T04:35:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:23 smithi129 ceph-mon[25657]: Deploying daemon node-exporter.b on smithi129 2024-03-26T04:35:23.395 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:23 smithi129 bash[41010]: Getting image source signatures 2024-03-26T04:35:23.395 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:23 smithi129 bash[41010]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-26T04:35:23.395 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:23 smithi129 bash[41010]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-26T04:35:23.395 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:23 smithi129 bash[41010]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-26T04:35:23.754 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:35:23.755 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":60,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","created":"2024-03-26T04:29:59.396661+0000","modified":"2024-03-26T04:35:00.336208+0000","last_up_change":"2024-03-26T04:34:44.985487+0000","last_in_change":"2024-03-26T04:34:28.871858+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:32:54.301748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"55282a5c-04c7-4f56-ae85-1f7b13034dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6803","nonce":857707498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6805","nonce":857707498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6809","nonce":857707498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6807","nonce":857707498}]},"public_addr":"172.21.15.17:6803/857707498","cluster_addr":"172.21.15.17:6805/857707498","heartbeat_back_addr":"172.21.15.17:6809/857707498","heartbeat_front_addr":"172.21.15.17:6807/857707498","state":["exists","up"]},{"osd":1,"uuid":"8b8f13d8-440b-4859-adac-d5dba48d4677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6811","nonce":3878069074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6813","nonce":3878069074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6817","nonce":3878069074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6815","nonce":3878069074}]},"public_addr":"172.21.15.17:6811/3878069074","cluster_addr":"172.21.15.17:6813/3878069074","heartbeat_back_addr":"172.21.15.17:6817/3878069074","heartbeat_front_addr":"172.21.15.17:6815/3878069074","state":["exists","up"]},{"osd":2,"uuid":"bbff43d2-2aec-401f-897d-afc6b7ab595b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6819","nonce":24179605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6821","nonce":24179605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6825","nonce":24179605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6823","nonce":24179605}]},"public_addr":"172.21.15.17:6819/24179605","cluster_addr":"172.21.15.17:6821/24179605","heartbeat_back_addr":"172.21.15.17:6825/24179605","heartbeat_front_addr":"172.21.15.17:6823/24179605","state":["exists","up"]},{"osd":3,"uuid":"c4aa4945-f8dd-4b25-9d97-61be1ca2faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6827","nonce":3671962567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6829","nonce":3671962567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6833","nonce":3671962567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6831","nonce":3671962567}]},"public_addr":"172.21.15.17:6827/3671962567","cluster_addr":"172.21.15.17:6829/3671962567","heartbeat_back_addr":"172.21.15.17:6833/3671962567","heartbeat_front_addr":"172.21.15.17:6831/3671962567","state":["exists","up"]},{"osd":4,"uuid":"6c36e2c8-de6d-48d4-b786-e710c5b729e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6801","nonce":195292081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6803","nonce":195292081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6807","nonce":195292081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6805","nonce":195292081}]},"public_addr":"172.21.15.129:6801/195292081","cluster_addr":"172.21.15.129:6803/195292081","heartbeat_back_addr":"172.21.15.129:6807/195292081","heartbeat_front_addr":"172.21.15.129:6805/195292081","state":["exists","up"]},{"osd":5,"uuid":"6449fee9-f734-462f-ad3a-f8eb7df4aed9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6809","nonce":1378709453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6811","nonce":1378709453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6815","nonce":1378709453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6813","nonce":1378709453}]},"public_addr":"172.21.15.129:6809/1378709453","cluster_addr":"172.21.15.129:6811/1378709453","heartbeat_back_addr":"172.21.15.129:6815/1378709453","heartbeat_front_addr":"172.21.15.129:6813/1378709453","state":["exists","up"]},{"osd":6,"uuid":"6badc639-2f90-40cf-b37e-fd17dbb0fca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6817","nonce":1036895989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6819","nonce":1036895989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6823","nonce":1036895989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6821","nonce":1036895989}]},"public_addr":"172.21.15.129:6817/1036895989","cluster_addr":"172.21.15.129:6819/1036895989","heartbeat_back_addr":"172.21.15.129:6823/1036895989","heartbeat_front_addr":"172.21.15.129:6821/1036895989","state":["exists","up"]},{"osd":7,"uuid":"d9c73505-35f9-41f0-a439-65dbb7122d6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6825","nonce":3889091653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6827","nonce":3889091653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6831","nonce":3889091653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6829","nonce":3889091653}]},"public_addr":"172.21.15.129:6825/3889091653","cluster_addr":"172.21.15.129:6827/3889091653","heartbeat_back_addr":"172.21.15.129:6831/3889091653","heartbeat_front_addr":"172.21.15.129:6829/3889091653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:04.414875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:27.268615+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:50.557826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:14.241162+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:35.508372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:58.049236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:34:21.309267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:34:43.727882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3562785501":"2024-03-27T04:35:00.336154+0000","172.21.15.17:6800/4071279932":"2024-03-27T04:35:00.336154+0000","172.21.15.17:6801/4126564882":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6801/4071279932":"2024-03-27T04:35:00.336154+0000","172.21.15.17:6800/4199384931":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1317244932":"2024-03-27T04:35:00.336154+0000","172.21.15.17:0/4101203109":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6801/4199384931":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/2717172247":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6800/4126564882":"2024-03-27T04:30:47.925915+0000","172.21.15.17:0/27415178":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1886501174":"2024-03-27T04:30:47.925915+0000","172.21.15.17:0/3729939084":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1272584175":"2024-03-27T04:35:00.336154+0000","172.21.15.17:0/2000177021":"2024-03-27T04:35:00.336154+0000","172.21.15.17:0/1619556253":"2024-03-27T04:30:25.908881+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:35:24.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:24 smithi017 ceph-mon[24814]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:24.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:24 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1180198240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:35:24.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:24 smithi017 ceph-mon[29581]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:24.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:24 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1180198240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:35:24.331 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T04:35:24.332 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:35:24.550 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:24.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:24 smithi129 ceph-mon[25657]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:24.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:24 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1180198240' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:35:25.145 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:24 smithi129 bash[41010]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-26T04:35:25.145 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:24 smithi129 bash[41010]: Writing manifest to image destination 2024-03-26T04:35:25.450 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 podman[41010]: 2024-03-26 04:35:25.192609751 +0000 UTC m=+2.320578428 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T04:35:25.451 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 podman[41010]: 2024-03-26 04:35:25.291319889 +0000 UTC m=+2.419288608 container create d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:35:25.451 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 podman[41010]: 2024-03-26 04:35:25.451322813 +0000 UTC m=+2.579291490 container init d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 podman[41010]: 2024-03-26 04:35:25.453889685 +0000 UTC m=+2.581858362 container start d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.457Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.457Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T04:35:25.896 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T04:35:25.897 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T04:35:25.898 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.458Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[41060]: ts=2024-03-26T04:35:25.459Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T04:35:25.899 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 bash[41010]: d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 2024-03-26T04:35:25.900 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:35:25 smithi129 systemd[1]: Started Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:35:26.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[24814]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[29581]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:25 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:26 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:26.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:26 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:26.265 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:35:26.265 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":60,"fsid":"67a15696-eb29-11ee-95d0-87774f69a715","created":"2024-03-26T04:29:59.396661+0000","modified":"2024-03-26T04:35:00.336208+0000","last_up_change":"2024-03-26T04:34:44.985487+0000","last_in_change":"2024-03-26T04:34:28.871858+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:32:54.301748+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"55282a5c-04c7-4f56-ae85-1f7b13034dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6803","nonce":857707498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6805","nonce":857707498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6809","nonce":857707498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":857707498},{"type":"v1","addr":"172.21.15.17:6807","nonce":857707498}]},"public_addr":"172.21.15.17:6803/857707498","cluster_addr":"172.21.15.17:6805/857707498","heartbeat_back_addr":"172.21.15.17:6809/857707498","heartbeat_front_addr":"172.21.15.17:6807/857707498","state":["exists","up"]},{"osd":1,"uuid":"8b8f13d8-440b-4859-adac-d5dba48d4677","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6811","nonce":3878069074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6813","nonce":3878069074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6817","nonce":3878069074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":3878069074},{"type":"v1","addr":"172.21.15.17:6815","nonce":3878069074}]},"public_addr":"172.21.15.17:6811/3878069074","cluster_addr":"172.21.15.17:6813/3878069074","heartbeat_back_addr":"172.21.15.17:6817/3878069074","heartbeat_front_addr":"172.21.15.17:6815/3878069074","state":["exists","up"]},{"osd":2,"uuid":"bbff43d2-2aec-401f-897d-afc6b7ab595b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6819","nonce":24179605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6821","nonce":24179605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6825","nonce":24179605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":24179605},{"type":"v1","addr":"172.21.15.17:6823","nonce":24179605}]},"public_addr":"172.21.15.17:6819/24179605","cluster_addr":"172.21.15.17:6821/24179605","heartbeat_back_addr":"172.21.15.17:6825/24179605","heartbeat_front_addr":"172.21.15.17:6823/24179605","state":["exists","up"]},{"osd":3,"uuid":"c4aa4945-f8dd-4b25-9d97-61be1ca2faf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6827","nonce":3671962567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6829","nonce":3671962567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6833","nonce":3671962567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":3671962567},{"type":"v1","addr":"172.21.15.17:6831","nonce":3671962567}]},"public_addr":"172.21.15.17:6827/3671962567","cluster_addr":"172.21.15.17:6829/3671962567","heartbeat_back_addr":"172.21.15.17:6833/3671962567","heartbeat_front_addr":"172.21.15.17:6831/3671962567","state":["exists","up"]},{"osd":4,"uuid":"6c36e2c8-de6d-48d4-b786-e710c5b729e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6801","nonce":195292081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6803","nonce":195292081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6807","nonce":195292081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":195292081},{"type":"v1","addr":"172.21.15.129:6805","nonce":195292081}]},"public_addr":"172.21.15.129:6801/195292081","cluster_addr":"172.21.15.129:6803/195292081","heartbeat_back_addr":"172.21.15.129:6807/195292081","heartbeat_front_addr":"172.21.15.129:6805/195292081","state":["exists","up"]},{"osd":5,"uuid":"6449fee9-f734-462f-ad3a-f8eb7df4aed9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6809","nonce":1378709453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6811","nonce":1378709453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6815","nonce":1378709453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":1378709453},{"type":"v1","addr":"172.21.15.129:6813","nonce":1378709453}]},"public_addr":"172.21.15.129:6809/1378709453","cluster_addr":"172.21.15.129:6811/1378709453","heartbeat_back_addr":"172.21.15.129:6815/1378709453","heartbeat_front_addr":"172.21.15.129:6813/1378709453","state":["exists","up"]},{"osd":6,"uuid":"6badc639-2f90-40cf-b37e-fd17dbb0fca6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6817","nonce":1036895989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6819","nonce":1036895989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6823","nonce":1036895989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1036895989},{"type":"v1","addr":"172.21.15.129:6821","nonce":1036895989}]},"public_addr":"172.21.15.129:6817/1036895989","cluster_addr":"172.21.15.129:6819/1036895989","heartbeat_back_addr":"172.21.15.129:6823/1036895989","heartbeat_front_addr":"172.21.15.129:6821/1036895989","state":["exists","up"]},{"osd":7,"uuid":"d9c73505-35f9-41f0-a439-65dbb7122d6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6825","nonce":3889091653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6827","nonce":3889091653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6831","nonce":3889091653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":3889091653},{"type":"v1","addr":"172.21.15.129:6829","nonce":3889091653}]},"public_addr":"172.21.15.129:6825/3889091653","cluster_addr":"172.21.15.129:6827/3889091653","heartbeat_back_addr":"172.21.15.129:6831/3889091653","heartbeat_front_addr":"172.21.15.129:6829/3889091653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:04.414875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:27.268615+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:32:50.557826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:14.241162+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:35.508372+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:33:58.049236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:34:21.309267+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:34:43.727882+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/3562785501":"2024-03-27T04:35:00.336154+0000","172.21.15.17:6800/4071279932":"2024-03-27T04:35:00.336154+0000","172.21.15.17:6801/4126564882":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6801/4071279932":"2024-03-27T04:35:00.336154+0000","172.21.15.17:6800/4199384931":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1317244932":"2024-03-27T04:35:00.336154+0000","172.21.15.17:0/4101203109":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6801/4199384931":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/2717172247":"2024-03-27T04:30:47.925915+0000","172.21.15.17:6800/4126564882":"2024-03-27T04:30:47.925915+0000","172.21.15.17:0/27415178":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1886501174":"2024-03-27T04:30:47.925915+0000","172.21.15.17:0/3729939084":"2024-03-27T04:30:25.908881+0000","172.21.15.17:0/1272584175":"2024-03-27T04:35:00.336154+0000","172.21.15.17:0/2000177021":"2024-03-27T04:35:00.336154+0000","172.21.15.17:0/1619556253":"2024-03-27T04:30:25.908881+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:35:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:26 smithi129 ceph-mon[25657]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:26 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:26 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:26.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:26 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:26.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:26 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:27.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:27 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:27.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:27 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3482394274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:35:27.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:27 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:27.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:27 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3482394274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:35:27.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:27 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:27.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:27 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3482394274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:35:27.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T04:35:27.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T04:35:27.375 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T04:35:27.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T04:35:27.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T04:35:27.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T04:35:27.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T04:35:27.377 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T04:35:27.837 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.838 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.838 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.842 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.842 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.843 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.843 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:27.847 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:28.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:28 smithi129 ceph-mon[25657]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:28.313 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:28 smithi017 ceph-mon[24814]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:28.314 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:28 smithi017 ceph-mon[29581]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:28.599 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:28] "GET /metrics HTTP/1.1" 200 191524 "" "Prometheus/2.33.4" 2024-03-26T04:35:29.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:29 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:29.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:29 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:29 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:30.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:30 smithi017 ceph-mon[29581]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:30.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:30 smithi017 ceph-mon[24814]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:30 smithi129 ceph-mon[25657]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:30.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:35:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:32 smithi129 ceph-mon[25657]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:32.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:32 smithi017 ceph-mon[24814]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:32.407 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:32 smithi017 ceph-mon[29581]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:34 smithi017 ceph-mon[24814]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:34.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:34 smithi017 ceph-mon[29581]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:34 smithi129 ceph-mon[25657]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:35.133 INFO:teuthology.orchestra.run.smithi017.stdout:111669149726 2024-03-26T04:35:35.134 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T04:35:36.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:36 smithi017 ceph-mon[24814]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:36 smithi017 ceph-mon[29581]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:36 smithi129 ceph-mon[25657]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:37.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:37 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:37.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:37 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:37.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:37 smithi129 ceph-mon[25657]: Deploying daemon alertmanager.a on smithi017 2024-03-26T04:35:37.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:37 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:37.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:37 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:37.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:37 smithi017 ceph-mon[24814]: Deploying daemon alertmanager.a on smithi017 2024-03-26T04:35:37.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:37 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:37.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:37 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:35:37.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:37 smithi017 ceph-mon[29581]: Deploying daemon alertmanager.a on smithi017 2024-03-26T04:35:38.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:38 smithi129 ceph-mon[25657]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:38.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:38 smithi017 ceph-mon[29581]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:38.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:38 smithi017 ceph-mon[24814]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:38.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:38] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2024-03-26T04:35:38.995 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:39.478 INFO:teuthology.orchestra.run.smithi017.stdout:236223201290 2024-03-26T04:35:39.478 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T04:35:40.117 INFO:teuthology.orchestra.run.smithi017.stdout:201863462929 2024-03-26T04:35:40.118 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T04:35:40.382 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:35:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:40 smithi129 ceph-mon[25657]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:40.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:40 smithi017 ceph-mon[24814]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:40.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:40 smithi017 ceph-mon[29581]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:41.055 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:42.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:42 smithi129 ceph-mon[25657]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:42.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:42 smithi017 ceph-mon[24814]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:42.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:42 smithi017 ceph-mon[29581]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:43.399 INFO:teuthology.orchestra.run.smithi017.stdout:133143986204 2024-03-26T04:35:43.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T04:35:43.945 INFO:teuthology.orchestra.run.smithi017.stdout:77309411364 2024-03-26T04:35:43.945 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T04:35:44.519 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:44 smithi017 ceph-mon[24814]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:44.520 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:44 smithi017 ceph-mon[29581]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:44 smithi129 ceph-mon[25657]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:45.783 INFO:teuthology.orchestra.run.smithi017.stdout:55834574887 2024-03-26T04:35:45.783 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T04:35:46.078 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:46 smithi017 ceph-mon[29581]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:46.079 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:46 smithi017 ceph-mon[24814]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:46 smithi129 ceph-mon[25657]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:46.823 INFO:teuthology.orchestra.run.smithi017.stdout:34359738414 2024-03-26T04:35:46.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T04:35:47.754 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:48.142 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:48 smithi017 ceph-mon[29581]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:48.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:48 smithi017 ceph-mon[24814]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:48.168 INFO:teuthology.orchestra.run.smithi017.stdout:167503724566 2024-03-26T04:35:48.168 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T04:35:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:48 smithi129 ceph-mon[25657]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:48.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:48] "GET /metrics HTTP/1.1" 200 191492 "" "Prometheus/2.33.4" 2024-03-26T04:35:49.041 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:49.041 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:49.042 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:50.354 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:50.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:35:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:50 smithi129 ceph-mon[25657]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:50.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:50 smithi017 ceph-mon[24814]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:50 smithi017 ceph-mon[29581]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:52 smithi129 ceph-mon[25657]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:52 smithi017 ceph-mon[24814]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:52 smithi017 ceph-mon[29581]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:54 smithi129 ceph-mon[25657]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:54.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:54 smithi017 ceph-mon[24814]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:54.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:54 smithi017 ceph-mon[29581]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:55.541 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:35:56.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:56 smithi017 ceph-mon[24814]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:56.325 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:56 smithi017 ceph-mon[29581]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:56 smithi129 ceph-mon[25657]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:57.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:57 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1680632500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:35:57.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:57 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1284502427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:35:57.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:57 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/642150143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:35:57.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:57 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1680632500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:35:57.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:57 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1284502427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:35:57.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:57 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/642150143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:35:57.275 INFO:teuthology.orchestra.run.smithi017.stdout:236223201294 2024-03-26T04:35:57.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:57 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1680632500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:35:57.297 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:57 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1284502427' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:35:57.297 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:57 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/642150143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:35:58.268 INFO:teuthology.orchestra.run.smithi017.stdout:201863462932 2024-03-26T04:35:58.269 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:58 smithi017 ceph-mon[24814]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:58.269 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:58 smithi017 ceph-mon[29581]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:58.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:58 smithi129 ceph-mon[25657]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:35:58.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:35:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:58] "GET /metrics HTTP/1.1" 200 191492 "" "Prometheus/2.33.4" 2024-03-26T04:35:58.713 INFO:teuthology.orchestra.run.smithi017.stdout:133143986206 2024-03-26T04:35:58.927 INFO:teuthology.orchestra.run.smithi017.stdout:34359738416 2024-03-26T04:35:59.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:59 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1727427582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:35:59.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:35:59 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1783036872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:35:59.340 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:59 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1727427582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:35:59.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:35:59 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1783036872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:35:59.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:59 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1727427582' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:35:59.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:35:59 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1783036872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:36:00.285 INFO:teuthology.orchestra.run.smithi017.stdout:55834574891 2024-03-26T04:36:00.286 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:00 smithi017 ceph-mon[24814]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:00.286 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:00 smithi017 ceph-mon[29581]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:00.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:00 smithi129 ceph-mon[25657]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:00.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:35:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:35:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:36:01.326 INFO:teuthology.orchestra.run.smithi017.stdout:111669149730 2024-03-26T04:36:01.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:01 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2511579763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:36:01.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:01 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2511579763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:36:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:01 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2511579763' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:36:01.901 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149726 got 111669149730 for osd.3 2024-03-26T04:36:01.901 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:02.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:02 smithi017 ceph-mon[24814]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:02.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:36:02.158 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:36:02.158 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:02 smithi017 ceph-mon[29581]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:02.158 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:36:02.158 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:36:02.391 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738416 for osd.0 2024-03-26T04:36:02.391 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:02 smithi129 ceph-mon[25657]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:36:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:36:02.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:02 smithi017 systemd[1]: Starting Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:36:03.113 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986206 for osd.4 2024-03-26T04:36:03.113 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:03.561 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462929 got 201863462932 for osd.6 2024-03-26T04:36:03.561 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:03.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574887 got 55834574891 for osd.1 2024-03-26T04:36:03.875 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:04.103 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:04 smithi017 ceph-mon[24814]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:04 smithi017 ceph-mon[29581]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:04 smithi129 ceph-mon[25657]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:05.278 INFO:teuthology.orchestra.run.smithi017.stdout:167503724570 2024-03-26T04:36:05.360 INFO:teuthology.orchestra.run.smithi017.stdout:77309411369 2024-03-26T04:36:05.811 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201290 got 236223201294 for osd.7 2024-03-26T04:36:05.811 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:06 smithi129 ceph-mon[25657]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:06 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2109787002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:36:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:06 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/914275485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:36:06.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:06 smithi017 ceph-mon[24814]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:06.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:06 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2109787002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:36:06.396 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:06 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/914275485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:36:06.396 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:06 smithi017 ceph-mon[29581]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:06.396 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:06 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2109787002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:36:06.396 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:06 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/914275485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:36:06.521 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411369 for osd.2 2024-03-26T04:36:06.521 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:07.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724566 got 167503724570 for osd.5 2024-03-26T04:36:07.004 DEBUG:teuthology.parallel:result is None 2024-03-26T04:36:07.004 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T04:36:07.004 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:36:07.341 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 podman[47507]: 2024-03-26 04:36:06.984715681 +0000 UTC m=+0.951034808 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-26T04:36:07.342 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 podman[47507]: 2024-03-26 04:36:07.093488403 +0000 UTC m=+1.059807513 volume create 96322e149a06b1a320e2ad3e998d56307b1b2b6d6221451ffe48db18f72fcee5 2024-03-26T04:36:07.342 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 podman[47507]: 2024-03-26 04:36:07.185673693 +0000 UTC m=+1.151992804 container create 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:07.662 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:36:07.786 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 podman[47507]: 2024-03-26 04:36:07.463411641 +0000 UTC m=+1.429730770 container init 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 podman[47507]: 2024-03-26 04:36:07.466060363 +0000 UTC m=+1.432379474 container start 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.476Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.476Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.477Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.479Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.503Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.504Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.506Z caller=main.go:518 msg=Listening address=:9093 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:07.506Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 bash[47507]: 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f 2024-03-26T04:36:07.787 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:07 smithi017 systemd[1]: Started Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:08 smithi129 ceph-mon[25657]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:08 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:08 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:08 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:08 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:36:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:08 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[29581]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[24814]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:36:08.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:08.736 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:36:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:08] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2024-03-26T04:36:09.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:09 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:36:09.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:09 smithi129 ceph-mon[25657]: Deploying daemon grafana.a on smithi129 2024-03-26T04:36:09.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:09 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:36:09.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:09 smithi017 ceph-mon[24814]: Deploying daemon grafana.a on smithi129 2024-03-26T04:36:09.437 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:09 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:36:09.437 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:09 smithi017 ceph-mon[29581]: Deploying daemon grafana.a on smithi129 2024-03-26T04:36:09.438 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:36:09.438 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-03-26T04:36:09.705 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:09 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:09.479Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000444321s 2024-03-26T04:36:10.133 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-26T04:36:09.018366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6295552,"data_stored":3510623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001953"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-26T04:35:01.106253+0000","last_change":"2024-03-26T04:34:49.008221+0000","last_active":"2024-03-26T04:35:01.106253+0000","last_peered":"2024-03-26T04:35:01.106253+0000","last_clean":"2024-03-26T04:35:01.106253+0000","last_became_active":"2024-03-26T04:34:49.007772+0000","last_became_peered":"2024-03-26T04:34:49.007772+0000","last_unstale":"2024-03-26T04:35:01.106253+0000","last_undegraded":"2024-03-26T04:35:01.106253+0000","last_fullsized":"2024-03-26T04:35:01.106253+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:32:54.419425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:32:54.419425+0000","last_clean_scrub_stamp":"2024-03-26T04:32:54.419425+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:01.165358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.432}]},{"osd":1,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.505},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004}]},{"osd":4,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.53700000000000003}]},{"osd":5,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.378},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.497}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.57799999999999996}]},{"osd":1,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.628}]},{"osd":3,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.54500000000000004}]},{"osd":5,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.64400000000000002}]},{"osd":7,"last update":"Tue Mar 26 04:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.67100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.33900000000000002,"15min":0.311},"min":{"1min":0.26800000000000002,"5min":0.154,"15min":0.154},"max":{"1min":0.48199999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.32400000000000001},"min":{"1min":0.26800000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.46200000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.371}]},{"osd":2,"last update":"Tue Mar 26 04:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.27200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51300000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.35699999999999998}]},{"osd":3,"last update":"Tue Mar 26 04:35:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.34200000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.54200000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.47799999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Tue Mar 26 04:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.56399999999999995},"min":{"1min":0.41199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.51900000000000002}]},{"osd":5,"last update":"Tue Mar 26 04:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57999999999999996}]},{"osd":6,"last update":"Tue Mar 26 04:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59799999999999998}]},{"osd":7,"last update":"Tue Mar 26 04:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 04:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.30599999999999999},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.46400000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.34499999999999997,"15min":0.32300000000000001},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59599999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Mar 26 04:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.32300000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59499999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Tue Mar 26 04:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.41399999999999998},"min":{"1min":0.313,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53900000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40300000000000002,"15min":0.40400000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Mar 26 04:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Mar 26 04:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.78000000000000003}]},{"osd":6,"last update":"Tue Mar 26 04:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.75800000000000001}]},{"osd":7,"last update":"Tue Mar 26 04:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.79200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39900000000000002,"15min":0.379},"min":{"1min":0.28999999999999998,"5min":0.254,"15min":0.254},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.34999999999999998}]},{"osd":1,"last update":"Tue Mar 26 04:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.376},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Mar 26 04:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.41899999999999998,"15min":0.42299999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.49399999999999999}]},{"osd":4,"last update":"Tue Mar 26 04:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56599999999999995}]},{"osd":5,"last update":"Tue Mar 26 04:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83599999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":0.65300000000000002}]},{"osd":6,"last update":"Tue Mar 26 04:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60399999999999998}]},{"osd":7,"last update":"Tue Mar 26 04:35:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":0.68899999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.52400000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.47799999999999998}]},{"osd":1,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58799999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45400000000000001,"15min":0.46100000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45400000000000001,"15min":0.46100000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70399999999999996,"5min":0.753,"15min":0.753},"last":0.50800000000000001}]},{"osd":4,"last update":"Tue Mar 26 04:35:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Mar 26 04:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66400000000000003}]},{"osd":6,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.69599999999999995}]},{"osd":7,"last update":"Tue Mar 26 04:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70799999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76600000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.57599999999999996}]},{"osd":1,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51700000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70599999999999996}]},{"osd":2,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65300000000000002}]},{"osd":5,"last update":"Tue Mar 26 04:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.437},"min":{"1min":0.40200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.56000000000000005}]},{"osd":6,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.443}]},{"osd":7,"last update":"Tue Mar 26 04:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.5},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42999999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5632,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739520,"statfs":{"total":95995035648,"available":95989268480,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.5,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71999999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71199999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51400000000000001,"15min":0.50900000000000001},"min":{"1min":0.39400000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.72599999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63}]},{"osd":2,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.438,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70999999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52400000000000002,"15min":0.51700000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.746,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.41099999999999998,"15min":0.40000000000000002},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60299999999999998}]},{"osd":6,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.70899999999999996}]},{"osd":7,"last update":"Tue Mar 26 04:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:36:10.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:36:10.360 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:36:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:10 smithi129 ceph-mon[25657]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:10.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:36:09 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:36:10.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:10 smithi017 ceph-mon[24814]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:10.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:10 smithi017 ceph-mon[29581]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:11 smithi129 ceph-mon[25657]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:36:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:11 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:11.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:11 smithi017 ceph-mon[24814]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:36:11.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:11 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:11.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:11 smithi017 ceph-mon[29581]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:36:11.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:11 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:36:12.189 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-03-26T04:36:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:12 smithi129 ceph-mon[25657]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:12.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:12 smithi017 ceph-mon[24814]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:12.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:12 smithi017 ceph-mon[29581]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:12.714 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-26T04:36:11.018787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6295552,"data_stored":3510623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002107"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-26T04:35:01.106253+0000","last_change":"2024-03-26T04:34:49.008221+0000","last_active":"2024-03-26T04:35:01.106253+0000","last_peered":"2024-03-26T04:35:01.106253+0000","last_clean":"2024-03-26T04:35:01.106253+0000","last_became_active":"2024-03-26T04:34:49.007772+0000","last_became_peered":"2024-03-26T04:34:49.007772+0000","last_unstale":"2024-03-26T04:35:01.106253+0000","last_undegraded":"2024-03-26T04:35:01.106253+0000","last_fullsized":"2024-03-26T04:35:01.106253+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:32:54.419425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:32:54.419425+0000","last_clean_scrub_stamp":"2024-03-26T04:32:54.419425+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:01.165358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.432}]},{"osd":1,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.505},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.54600000000000004}]},{"osd":4,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.53700000000000003}]},{"osd":5,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55900000000000005}]},{"osd":6,"last update":"Tue Mar 26 04:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.378},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.497}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.57799999999999996}]},{"osd":1,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.48999999999999999}]},{"osd":2,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.628}]},{"osd":3,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61399999999999999}]},{"osd":4,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.54500000000000004}]},{"osd":5,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.64400000000000002}]},{"osd":7,"last update":"Tue Mar 26 04:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.67100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.33900000000000002,"15min":0.311},"min":{"1min":0.26800000000000002,"5min":0.154,"15min":0.154},"max":{"1min":0.48199999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.32400000000000001},"min":{"1min":0.26800000000000002,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":0.46200000000000002,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.371}]},{"osd":2,"last update":"Tue Mar 26 04:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.27200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.51300000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38600000000000001,"15min":0.39100000000000001},"min":{"1min":0.27700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.49199999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.35699999999999998}]},{"osd":3,"last update":"Tue Mar 26 04:35:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.44600000000000001,"15min":0.45200000000000001},"min":{"1min":0.34200000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.54200000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.47799999999999998,"15min":0.48399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Tue Mar 26 04:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.56399999999999995},"min":{"1min":0.41199999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.51900000000000002}]},{"osd":5,"last update":"Tue Mar 26 04:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57999999999999996}]},{"osd":6,"last update":"Tue Mar 26 04:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59799999999999998}]},{"osd":7,"last update":"Tue Mar 26 04:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.53800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 04:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.32800000000000001,"15min":0.30599999999999999},"min":{"1min":0.24299999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.46400000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.34499999999999997,"15min":0.32300000000000001},"min":{"1min":0.23300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.59599999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.36599999999999999}]},{"osd":2,"last update":"Tue Mar 26 04:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.32300000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.39100000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59499999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Tue Mar 26 04:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.41299999999999998,"15min":0.41399999999999998},"min":{"1min":0.313,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.53900000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.40300000000000002,"15min":0.40400000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Mar 26 04:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73299999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Mar 26 04:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.60099999999999998,"15min":0.60299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.78000000000000003}]},{"osd":6,"last update":"Tue Mar 26 04:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.75800000000000001}]},{"osd":7,"last update":"Tue Mar 26 04:35:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.79200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.30199999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39900000000000002,"15min":0.379},"min":{"1min":0.28999999999999998,"5min":0.254,"15min":0.254},"max":{"1min":1.907,"5min":1.907,"15min":1.907},"last":0.34999999999999998}]},{"osd":1,"last update":"Tue Mar 26 04:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.39200000000000002,"15min":0.376},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Mar 26 04:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.41899999999999998,"15min":0.42299999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.621,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.49399999999999999}]},{"osd":4,"last update":"Tue Mar 26 04:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.57699999999999996,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56599999999999995}]},{"osd":5,"last update":"Tue Mar 26 04:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83599999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":0.65300000000000002}]},{"osd":6,"last update":"Tue Mar 26 04:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60399999999999998}]},{"osd":7,"last update":"Tue Mar 26 04:35:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":2.0830000000000002,"5min":2.0830000000000002,"15min":2.0830000000000002},"last":0.68899999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.39400000000000002,"15min":0.39700000000000002},"min":{"1min":0.247,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57299999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.40899999999999997,"15min":0.41399999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.52400000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.47799999999999998}]},{"osd":1,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58799999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45400000000000001,"15min":0.46100000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45400000000000001,"15min":0.46100000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60099999999999998,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.433,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70399999999999996,"5min":0.753,"15min":0.753},"last":0.50800000000000001}]},{"osd":4,"last update":"Tue Mar 26 04:35:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Mar 26 04:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66400000000000003}]},{"osd":6,"last update":"Tue Mar 26 04:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.69599999999999995}]},{"osd":7,"last update":"Tue Mar 26 04:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70799999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.76600000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50800000000000001,"15min":0.50600000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.57599999999999996}]},{"osd":1,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.51700000000000002,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.74399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51200000000000001,"15min":0.50600000000000001},"min":{"1min":0.309,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70599999999999996}]},{"osd":2,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55000000000000004,"15min":0.54700000000000004},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Mar 26 04:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65300000000000002}]},{"osd":5,"last update":"Tue Mar 26 04:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45100000000000001,"15min":0.437},"min":{"1min":0.40200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.56000000000000005}]},{"osd":6,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.443}]},{"osd":7,"last update":"Tue Mar 26 04:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.5},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42999999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5632,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739520,"statfs":{"total":95995035648,"available":95989268480,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.51700000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.5,"15min":0.49399999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71999999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.433,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71199999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51400000000000001,"15min":0.50900000000000001},"min":{"1min":0.39400000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.72599999999999998,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.63}]},{"osd":2,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.438,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70999999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52400000000000002,"15min":0.51700000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.746,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Mar 26 04:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.41099999999999998,"15min":0.40000000000000002},"min":{"1min":0.30399999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60299999999999998}]},{"osd":6,"last update":"Tue Mar 26 04:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.70899999999999996}]},{"osd":7,"last update":"Tue Mar 26 04:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:36:12.715 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T04:36:12.715 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T04:36:12.716 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T04:36:12.716 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T04:36:12.937 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:36:13.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:13 smithi017 ceph-mon[24814]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:36:13.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:13 smithi017 ceph-mon[29581]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:36:13.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:13 smithi129 ceph-mon[25657]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:36:14.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:14 smithi017 ceph-mon[24814]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:14.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:14 smithi017 ceph-mon[29581]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:14 smithi129 ceph-mon[25657]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-03-26T04:36:14.629 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T04:36:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:15 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1185413359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:36:15.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:15 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1185413359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:36:15.312 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T04:36:15.312 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T04:36:15.313 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T04:36:15.322 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-03-26T04:36:15.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-03-26T04:36:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:15 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1185413359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:36:16.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:16 smithi017 ceph-mon[24814]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:16.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:16 smithi017 ceph-mon[29581]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:16 smithi129 ceph-mon[25657]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:17.705 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:17 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:17.481Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00182241s 2024-03-26T04:36:17.867 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T04:36:17.874 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-03-26T04:36:17.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-26T04:36:18.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:18 smithi017 ceph-mon[24814]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:18.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:18 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' 2024-03-26T04:36:18.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:18 smithi017 ceph-mon[29581]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:18.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:18 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' 2024-03-26T04:36:18.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:18 smithi129 ceph-mon[25657]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:18.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:18 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' 2024-03-26T04:36:18.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:36:18 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:18] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-03-26T04:36:19.972 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:36:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:36:20.561 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-mon[25657]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:20.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 systemd[1]: Starting Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:36:20.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:20 smithi017 ceph-mon[24814]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:20.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:20 smithi017 ceph-mon[29581]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:20.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 podman[41739]: 2024-03-26 04:36:20.461630705 +0000 UTC m=+0.017084093 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T04:36:20.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 podman[41739]: 2024-03-26 04:36:20.559242431 +0000 UTC m=+0.114695805 container create 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, distribution-scope=public, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.openshift.tags=base rhel8, version=8.5, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, name=ubi8, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64) 2024-03-26T04:36:20.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 podman[41739]: 2024-03-26 04:36:20.762310673 +0000 UTC m=+0.317764057 container init 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, maintainer=Paul Cuzner , name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-type=git, description=Ceph Grafana Container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container) 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 podman[41739]: 2024-03-26 04:36:20.765109953 +0000 UTC m=+0.320563330 container start 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, vcs-type=git, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.buildah.version=1.24.2, version=8.5, release=236.1648460182, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, description=Ceph Grafana Container) 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-26T04:36:20.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="App mode production" logger=settings 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-26T04:36:20.811 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-26T04:36:20.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T04:36:20.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-26T04:36:20.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-26T04:36:21.059 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-26T04:36:21.059 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-26T04:36:21.059 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-26T04:36:21.060 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-26T04:36:21.061 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 bash[41739]: 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 systemd[1]: Started Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-26T04:36:21.062 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T04:36:21.063 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-26T04:36:21.064 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-26T04:36:21.065 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T04:36:21.066 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T04:36:21.067 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-26T04:36:21.068 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-26T04:36:21.069 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:20+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-26T04:36:21.070 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-26T04:36:21.071 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-26T04:36:21.072 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-26T04:36:21.073 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-26T04:36:21.074 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T04:36:21.294 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:36:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "bb32a8c6-6ea5-452f-85fb-b3ceb475291f", 2024-03-26T04:36:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "r", 2024-03-26T04:36:21.294 INFO:teuthology.orchestra.run.smithi017.stdout: "current_period": "514ab2b8-3231-47d8-889f-0bfbf125c13d", 2024-03-26T04:36:21.295 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2024-03-26T04:36:21.295 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3107015721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3107015721' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:36:21.368 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-mon[25657]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-26T04:36:21.369 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-26T04:36:21.370 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T04:36:21.371 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-26T04:36:21.372 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-26T04:36:21.373 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T04:36:21.374 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-26T04:36:21.375 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T04:36:21.376 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-26T04:36:21.377 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-26T04:36:21.378 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-26T04:36:21.379 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=446.196116ms 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-26T04:36:21.380 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-26T04:36:21.381 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3107015721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3107015721' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:36:21.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[24814]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3107015721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3107015721' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:36:21.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:21 smithi017 ceph-mon[29581]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:36:21.645 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:36:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:36:21+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-26T04:36:22.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-26T04:36:22.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:22 smithi129 ceph-mon[25657]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:22.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:22 smithi129 ceph-mon[25657]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:36:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:22 smithi017 ceph-mon[29581]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:22 smithi017 ceph-mon[29581]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:36:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:22 smithi017 ceph-mon[24814]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:22.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:22 smithi017 ceph-mon[24814]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "7a419524-6c04-4fe4-97d4-2ef4eedadf84", 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "default", 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": "true", 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "", 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [], 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [], 2024-03-26T04:36:24.196 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "", 2024-03-26T04:36:24.197 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "bb32a8c6-6ea5-452f-85fb-b3ceb475291f", 2024-03-26T04:36:24.197 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2024-03-26T04:36:24.197 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2024-03-26T04:36:24.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:24.197 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:36:24.523 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:24 smithi129 ceph-mon[25657]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T04:36:24.523 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:24 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:24.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:24 smithi017 ceph-mon[24814]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T04:36:24.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:24 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:24.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:24 smithi017 ceph-mon[29581]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T04:36:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:24 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:24.799 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-26T04:36:26.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:26 smithi017 ceph-mon[24814]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T04:36:26.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:26 smithi017 ceph-mon[29581]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T04:36:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:26 smithi129 ceph-mon[25657]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "97846378-9364-4a79-b573-2830278f214f", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "z", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "z.rgw.meta:root", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "z.rgw.control", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "z.rgw.log", 2024-03-26T04:36:27.643 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "z.rgw.otp", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "", 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "" 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-03-26T04:36:27.644 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "bb32a8c6-6ea5-452f-85fb-b3ceb475291f", 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-26T04:36:27.645 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:36:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-mon[29581]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-26T04:36:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:28.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-mon[24814]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-26T04:36:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:28.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:28.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-26T04:36:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:28 smithi129 ceph-mon[25657]: pgmap v49: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-26T04:36:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:28 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:28 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:28.751 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:36:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:28] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-03-26T04:36:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:29 smithi129 ceph-mon[25657]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:36:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:29 smithi129 ceph-mon[25657]: Reconfiguring daemon alertmanager.a on smithi017 2024-03-26T04:36:29.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:29 smithi017 ceph-mon[24814]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:36:29.406 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:29 smithi017 ceph-mon[24814]: Reconfiguring daemon alertmanager.a on smithi017 2024-03-26T04:36:29.406 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:29 smithi017 ceph-mon[29581]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:36:29.406 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:29 smithi017 ceph-mon[29581]: Reconfiguring daemon alertmanager.a on smithi017 2024-03-26T04:36:30.319 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:30 smithi017 ceph-mon[24814]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-26T04:36:30.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:30 smithi017 ceph-mon[29581]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-26T04:36:30.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:36:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:36:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:30 smithi129 ceph-mon[25657]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-26T04:36:31.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:31 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:36:32.025 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:31 smithi017 bash[49288]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T04:36:32.026 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[47561]: level=info ts=2024-03-26T04:36:31.908Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:36:32.026 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:31 smithi017 podman[49303]: 2024-03-26 04:36:31.928477881 +0000 UTC m=+0.137807602 container died 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:32.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:32 smithi017 ceph-mon[24814]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-26T04:36:32.299 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:32 smithi017 ceph-mon[29581]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-26T04:36:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:32 smithi129 ceph-mon[25657]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-26T04:36:32.652 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 podman[49303]: 2024-03-26 04:36:32.300059855 +0000 UTC m=+0.509389582 container cleanup 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:32.653 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 bash[49303]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:36:32.653 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 bash[49347]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T04:36:32.945 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 podman[49335]: 2024-03-26 04:36:32.845051312 +0000 UTC m=+0.912332445 container remove 1b35a1ca2abe6ef0a09663edc85ea29a84286223ce6af14d3d5406d95256815f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:33.204 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 podman[49335]: 2024-03-26 04:36:32.94590809 +0000 UTC m=+1.013189223 volume remove 96322e149a06b1a320e2ad3e998d56307b1b2b6d6221451ffe48db18f72fcee5 2024-03-26T04:36:33.205 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:36:33.205 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:36:33.205 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:32 smithi017 systemd[1]: Starting Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:36:33.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 podman[49473]: 2024-03-26 04:36:33.322854841 +0000 UTC m=+0.079710907 volume create d9cb0cc23e77d2bd67ce0a58e042adc1b583f6effd13e609cbac724ac9ee73e5 2024-03-26T04:36:33.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 podman[49473]: 2024-03-26 04:36:33.261076235 +0000 UTC m=+0.017932309 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-26T04:36:33.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 podman[49473]: 2024-03-26 04:36:33.414921295 +0000 UTC m=+0.171777354 container create f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:33.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 podman[49473]: 2024-03-26 04:36:33.725928956 +0000 UTC m=+0.482785019 container init f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 podman[49473]: 2024-03-26 04:36:33.728980396 +0000 UTC m=+0.485836454 container start f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.739Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.739Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.740Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.741Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.767Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.767Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.769Z caller=main.go:518 msg=Listening address=:9093 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:33.769Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 bash[49473]: f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 2024-03-26T04:36:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:33 smithi017 systemd[1]: Started Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:36:34.239 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:34 smithi129 ceph-mon[25657]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 665 B/s rd, 761 B/s wr, 1 op/s 2024-03-26T04:36:34.239 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:34 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:34 smithi017 ceph-mon[24814]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 665 B/s rd, 761 B/s wr, 1 op/s 2024-03-26T04:36:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:34 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:34.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:34 smithi017 ceph-mon[29581]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 665 B/s rd, 761 B/s wr, 1 op/s 2024-03-26T04:36:34.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:34 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:35.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:35 smithi129 ceph-mon[25657]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:36:35.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:35 smithi129 ceph-mon[25657]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:36:35.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:35 smithi129 ceph-mon[25657]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:36:35.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:35 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:36:35.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[29581]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[29581]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[29581]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[24814]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[24814]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[24814]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:36:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:36:36.099 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:35.742Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000349658s 2024-03-26T04:36:36.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-mon[25657]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 819 B/s wr, 1 op/s 2024-03-26T04:36:36.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-26T04:36:36.350 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-mon[25657]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:36:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:36 smithi017 ceph-mon[24814]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 819 B/s wr, 1 op/s 2024-03-26T04:36:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:36 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-26T04:36:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:36 smithi017 ceph-mon[24814]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:36:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:36 smithi017 ceph-mon[29581]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 819 B/s wr, 1 op/s 2024-03-26T04:36:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:36 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-26T04:36:36.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:36 smithi017 ceph-mon[29581]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:36:36.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 systemd[1]: Stopping Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:36:36.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 bash[42433]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:36:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:37 smithi129 ceph-mon[25657]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:36:37.180 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:37 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:36:37.180 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.903Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:36:37.180 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:36:37.180 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:36:37.180 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.904Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.905Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.905Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[40751]: ts=2024-03-26T04:36:36.905Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T04:36:37.181 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:36 smithi129 podman[42446]: 2024-03-26 04:36:36.914267711 +0000 UTC m=+0.108071520 container died 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:37.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:37 smithi017 ceph-mon[24814]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:36:37.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:37 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:36:37.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:37 smithi017 ceph-mon[29581]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:36:37.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:37 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:36:37.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 podman[42446]: 2024-03-26 04:36:37.370987518 +0000 UTC m=+0.564791360 container cleanup 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:37.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 bash[42446]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:36:37.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 bash[42473]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:36:37.946 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 podman[42460]: 2024-03-26 04:36:37.655433058 +0000 UTC m=+0.742775251 container remove 29fc3d8c976cdbf50f2025047a5b06fc7054c2846d595cf72a5bcea425d71bf4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:37.946 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T04:36:37.946 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 systemd[1]: Stopped Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:36:37.947 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 systemd[1]: Starting Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:36:37.947 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 podman[42539]: 2024-03-26 04:36:37.864812602 +0000 UTC m=+0.098699158 container create faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:37.947 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:37 smithi129 podman[42539]: 2024-03-26 04:36:37.784930462 +0000 UTC m=+0.018817015 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T04:36:38.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-mon[25657]: pgmap v57: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-26T04:36:38.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-26T04:36:38.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-mon[25657]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:36:38.398 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 podman[42539]: 2024-03-26 04:36:38.041807993 +0000 UTC m=+0.275694546 container init faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 podman[42539]: 2024-03-26 04:36:38.044222172 +0000 UTC m=+0.278108726 container start faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.067Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.067Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.067Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.067Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi129 (none))" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.067Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.067Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.069Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.069Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.071Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.072Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.072Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.439µs 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.072Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:36:38.399 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 bash[42539]: faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 systemd[1]: Started Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.270Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.270Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.270Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.397µs wal_replay_duration=197.645913ms total_replay_duration=197.703194ms 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.271Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.271Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.271Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.286Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.615932ms db_storage=760ns remote_storage=1.24µs web_handler=317ns query_engine=913ns scrape=391.792µs scrape_sd=22.775µs notify=27.554µs notify_sd=12.441µs rules=13.947481ms 2024-03-26T04:36:38.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:36:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:36:38.286Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T04:36:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:38 smithi017 ceph-mon[24814]: pgmap v57: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-26T04:36:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:38 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-26T04:36:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:38 smithi017 ceph-mon[24814]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:36:38.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:38 smithi017 ceph-mon[29581]: pgmap v57: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-26T04:36:38.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:38 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1499164152' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-26T04:36:38.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:38 smithi017 ceph-mon[29581]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:36:39.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:36:39.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:36:39.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.129:9095"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.129:9095"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.129:3000"}]: dispatch 2024-03-26T04:36:39.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.129:3000"}]: dispatch 2024-03-26T04:36:39.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:39.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:39 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.129:9095"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.129:9095"}]: dispatch 2024-03-26T04:36:39.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.129:3000"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.129:3000"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:36:39.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.129:9095"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.129:9095"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.129:3000"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.129:3000"}]: dispatch 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:39.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:39.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:39 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:36:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:40 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T04:36:40.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:36:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:36:40.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:36:40.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:40 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "31ce9aec-bff3-4ff7-bd1e-91b967250e5e", 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "514ab2b8-3231-47d8-889f-0bfbf125c13d", 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "31ce9aec-bff3-4ff7-bd1e-91b967250e5e", 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "7a419524-6c04-4fe4-97d4-2ef4eedadf84", 2024-03-26T04:36:41.172 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "default", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": "true", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "97846378-9364-4a79-b573-2830278f214f", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "97846378-9364-4a79-b573-2830278f214f", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "z", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": "false", 2024-03-26T04:36:41.173 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": "false", 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": "false", 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": "true", 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "" 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2024-03-26T04:36:41.174 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "bb32a8c6-6ea5-452f-85fb-b3ceb475291f", 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "97846378-9364-4a79-b573-2830278f214f", 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 3869662401 2024-03-26T04:36:41.175 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "7a419524-6c04-4fe4-97d4-2ef4eedadf84", 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "97846378-9364-4a79-b573-2830278f214f", 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2024-03-26T04:36:41.176 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-03-26T04:36:41.177 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "bb32a8c6-6ea5-452f-85fb-b3ceb475291f", 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "r", 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2024-03-26T04:36:41.178 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:36:41.947 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-26T04:36:42.108 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:42.109 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:41 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-26T04:36:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:36:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2395298344' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:42.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:41 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:43.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:36:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:36:43.745Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003242406s 2024-03-26T04:36:44.098 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled rgw.foo update... 2024-03-26T04:36:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:44 smithi129 ceph-mon[25657]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T04:36:44.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:44 smithi017 ceph-mon[29581]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T04:36:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:44 smithi017 ceph-mon[24814]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-26T04:36:44.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch apply rgw smpl' 2024-03-26T04:36:45.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:45 smithi017 ceph-mon[24814]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:36:45.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:45 smithi017 ceph-mon[24814]: Saving service rgw.foo spec with placement count:2 2024-03-26T04:36:45.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:45 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:45.368 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:45 smithi017 ceph-mon[29581]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:36:45.368 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:45 smithi017 ceph-mon[29581]: Saving service rgw.foo spec with placement count:2 2024-03-26T04:36:45.369 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:45 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:45.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:45 smithi129 ceph-mon[25657]: from='client.14700 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:36:45.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:45 smithi129 ceph-mon[25657]: Saving service rgw.foo spec with placement count:2 2024-03-26T04:36:45.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:45 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.365 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.366 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:46 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:36:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:46.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:46 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:46.891 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled rgw.smpl update... 2024-03-26T04:36:47.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:47 smithi129 ceph-mon[25657]: Saving service rgw.foo spec with placement count:2 2024-03-26T04:36:47.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:47 smithi129 ceph-mon[25657]: Deploying daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:36:47.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:47 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:47 smithi017 ceph-mon[24814]: Saving service rgw.foo spec with placement count:2 2024-03-26T04:36:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:47 smithi017 ceph-mon[24814]: Deploying daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:36:47.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:47 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:47.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:47 smithi017 ceph-mon[29581]: Saving service rgw.foo spec with placement count:2 2024-03-26T04:36:47.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:47 smithi017 ceph-mon[29581]: Deploying daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:36:47.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:47 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:47.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph osd pool create foo' 2024-03-26T04:36:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:48 smithi129 ceph-mon[25657]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:36:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:48 smithi129 ceph-mon[25657]: Saving service rgw.smpl spec with placement count:2 2024-03-26T04:36:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:48 smithi129 ceph-mon[25657]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T04:36:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-mon[24814]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:36:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-mon[24814]: Saving service rgw.smpl spec with placement count:2 2024-03-26T04:36:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-mon[24814]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T04:36:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-mon[29581]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:36:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-mon[29581]: Saving service rgw.smpl spec with placement count:2 2024-03-26T04:36:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-mon[29581]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-26T04:36:48.716 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:36:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:48] "GET /metrics HTTP/1.1" 200 196837 "" "Prometheus/2.33.4" 2024-03-26T04:36:50.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:50 smithi017 ceph-mon[24814]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-26T04:36:50.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:50 smithi017 ceph-mon[29581]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-26T04:36:50.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:36:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:36:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:50 smithi129 ceph-mon[25657]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-26T04:36:51.120 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'foo' created 2024-03-26T04:36:51.356 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:51 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4251386732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T04:36:51.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:51 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:51.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:51 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:51.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:51 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:51.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:51 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:51.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:51 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4251386732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4251386732' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:51.402 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:51.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:51.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:51.403 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:51 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:51.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'rbd pool init foo' 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[24814]: Deploying daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[24814]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4251386732' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[24814]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[29581]: Deploying daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[29581]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4251386732' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-26T04:36:52.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:52 smithi017 ceph-mon[29581]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:36:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:52 smithi129 ceph-mon[25657]: Deploying daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:36:52.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:52 smithi129 ceph-mon[25657]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 819 B/s rd, 921 B/s wr, 2 op/s 2024-03-26T04:36:52.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:52 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4251386732' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-26T04:36:52.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:52 smithi129 ceph-mon[25657]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:36:53.364 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:53 smithi017 ceph-mon[29581]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:36:53.364 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:53 smithi017 ceph-mon[24814]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:36:53.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:53 smithi129 ceph-mon[25657]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 4.1 KiB/s wr, 196 op/s 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2506863332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T04:36:54.125 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:54 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 4.1 KiB/s wr, 196 op/s 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2506863332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: pgmap v72: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 4.1 KiB/s wr, 196 op/s 2024-03-26T04:36:54.430 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:36:54.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:54.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:54.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:54.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2506863332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T04:36:54.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:54 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T04:36:55.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:55 smithi129 ceph-mon[25657]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-26T04:36:55.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:55 smithi129 ceph-mon[25657]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:36:55.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:55 smithi017 ceph-mon[24814]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-26T04:36:55.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:55 smithi017 ceph-mon[24814]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:36:55.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:55 smithi017 ceph-mon[29581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-26T04:36:55.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:55 smithi017 ceph-mon[29581]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:36:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:56 smithi017 ceph-mon[24814]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 183 KiB/s rd, 8.2 KiB/s wr, 392 op/s 2024-03-26T04:36:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:56 smithi017 ceph-mon[24814]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:36:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:56 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:56.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:56 smithi017 ceph-mon[29581]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 183 KiB/s rd, 8.2 KiB/s wr, 392 op/s 2024-03-26T04:36:56.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:56 smithi017 ceph-mon[29581]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:36:56.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:56 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:56.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:56 smithi129 ceph-mon[25657]: pgmap v75: 161 pgs: 32 creating+peering, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 183 KiB/s rd, 8.2 KiB/s wr, 392 op/s 2024-03-26T04:36:56.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:56 smithi129 ceph-mon[25657]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:36:56.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:56 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:56.881 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-26T04:36:57.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:57 smithi017 ceph-mon[29581]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:36:57.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:57 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:57.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:57 smithi017 ceph-mon[24814]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:36:57.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:57 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:57.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:57 smithi129 ceph-mon[25657]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:36:57.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:57 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 260 KiB/s rd, 767 B/s wr, 397 op/s 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: Checking dashboard <-> RGW credentials 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:58.547 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:58.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:58 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:58.586 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:58] "GET /metrics HTTP/1.1" 200 196837 "" "Prometheus/2.33.4" 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 260 KiB/s rd, 767 B/s wr, 397 op/s 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: Checking dashboard <-> RGW credentials 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.587 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 260 KiB/s rd, 767 B/s wr, 397 op/s 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: Checking dashboard <-> RGW credentials 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:58.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:58 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:59.087 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled iscsi.foo update... 2024-03-26T04:36:59.399 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:59 smithi017 ceph-mon[29581]: Saving service rgw.smpl spec with placement count:2 2024-03-26T04:36:59.399 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:59 smithi017 ceph-mon[29581]: Deploying daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:36:59.399 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:36:59 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:59.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:59 smithi017 ceph-mon[24814]: Saving service rgw.smpl spec with placement count:2 2024-03-26T04:36:59.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:59 smithi017 ceph-mon[24814]: Deploying daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:36:59.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:36:59 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:59 smithi129 ceph-mon[25657]: Saving service rgw.smpl spec with placement count:2 2024-03-26T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:59 smithi129 ceph-mon[25657]: Deploying daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:36:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:36:59 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:00.242 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:36:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:36:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:37:00.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'sleep 120' 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[29581]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 520 B/s wr, 269 op/s 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[29581]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[29581]: Saving service iscsi.foo spec with placement count:1 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[29581]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[24814]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 520 B/s wr, 269 op/s 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[24814]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[24814]: Saving service iscsi.foo spec with placement count:1 2024-03-26T04:37:00.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:00 smithi017 ceph-mon[24814]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:37:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:00 smithi129 ceph-mon[25657]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 520 B/s wr, 269 op/s 2024-03-26T04:37:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:00 smithi129 ceph-mon[25657]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:37:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:00 smithi129 ceph-mon[25657]: Saving service iscsi.foo spec with placement count:1 2024-03-26T04:37:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:00 smithi129 ceph-mon[25657]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:37:01.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:37:01.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:37:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-03-26T04:37:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-26T04:37:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-26T04:37:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-26T04:37:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:37:01.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:37:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:02 smithi129 ceph-mon[25657]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 511 B/s wr, 265 op/s 2024-03-26T04:37:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-03-26T04:37:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-26T04:37:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-26T04:37:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-26T04:37:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:02 smithi129 ceph-mon[25657]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:37:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[24814]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 511 B/s wr, 265 op/s 2024-03-26T04:37:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-03-26T04:37:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-26T04:37:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-26T04:37:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-26T04:37:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[24814]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:37:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[29581]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 511 B/s wr, 265 op/s 2024-03-26T04:37:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-03-26T04:37:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-26T04:37:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-26T04:37:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-26T04:37:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:02 smithi017 ceph-mon[29581]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:37:03.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[29581]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[24814]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:37:03.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:03.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:03 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:03.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:03 smithi129 ceph-mon[25657]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:37:03.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:03.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:37:03.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:37:03.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:03.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:03 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:04.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:04 smithi129 ceph-mon[25657]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 26 op/s 2024-03-26T04:37:04.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:04 smithi129 ceph-mon[25657]: Deploying daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:37:04.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:04 smithi017 ceph-mon[24814]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 26 op/s 2024-03-26T04:37:04.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:04 smithi017 ceph-mon[24814]: Deploying daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:37:04.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:04 smithi017 ceph-mon[29581]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 26 op/s 2024-03-26T04:37:04.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:04 smithi017 ceph-mon[29581]: Deploying daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:37:06.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:06 smithi129 ceph-mon[25657]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 26 op/s 2024-03-26T04:37:06.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:06 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:06.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:06 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:06.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:06 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:06.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[29581]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 26 op/s 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[24814]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 26 op/s 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:06.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:06 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:07.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:07 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:07.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:07 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:07.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:07 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:08 smithi129 ceph-mon[25657]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 269 KiB/s rd, 583 B/s wr, 409 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:08 smithi017 ceph-mon[24814]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 269 KiB/s rd, 583 B/s wr, 409 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:08 smithi017 ceph-mon[29581]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 269 KiB/s rd, 583 B/s wr, 409 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:08.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:37:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:08] "GET /metrics HTTP/1.1" 200 208017 "" "Prometheus/2.33.4" 2024-03-26T04:37:09.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:37:09.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:09 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:37:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:37:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:10.233 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:37:09 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:37:10.580 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[24814]: Checking dashboard <-> RGW credentials 2024-03-26T04:37:10.580 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[24814]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 236 KiB/s rd, 511 B/s wr, 358 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:10.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[24814]: Checking pool "foo" exists for service iscsi.foo 2024-03-26T04:37:10.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[24814]: Deploying daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:37:10.581 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[29581]: Checking dashboard <-> RGW credentials 2024-03-26T04:37:10.581 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[29581]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 236 KiB/s rd, 511 B/s wr, 358 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:10.581 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[29581]: Checking pool "foo" exists for service iscsi.foo 2024-03-26T04:37:10.581 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:10 smithi017 ceph-mon[29581]: Deploying daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:37:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:10 smithi129 ceph-mon[25657]: Checking dashboard <-> RGW credentials 2024-03-26T04:37:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:10 smithi129 ceph-mon[25657]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 236 KiB/s rd, 511 B/s wr, 358 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:10 smithi129 ceph-mon[25657]: Checking pool "foo" exists for service iscsi.foo 2024-03-26T04:37:10.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:10 smithi129 ceph-mon[25657]: Deploying daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:37:12.547 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:12 smithi017 ceph-mon[24814]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 418 B/s wr, 293 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:12.548 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:12 smithi017 ceph-mon[24814]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-26T04:37:12.548 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:12 smithi017 ceph-mon[29581]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 418 B/s wr, 293 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:12.548 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:12 smithi017 ceph-mon[29581]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-26T04:37:12.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:12 smithi129 ceph-mon[25657]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 418 B/s wr, 293 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:12.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:12 smithi129 ceph-mon[25657]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-26T04:37:13.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:13.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:13.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:13.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1157471972' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:37:13.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2467035683' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3562785501"}]: dispatch 2024-03-26T04:37:13.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:13.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:13.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:13.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1157471972' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:37:13.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2467035683' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3562785501"}]: dispatch 2024-03-26T04:37:13.707 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:37:13.528Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:37:13.707 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:13 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:37:13.528Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:13 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1157471972' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:37:13.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:13 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2467035683' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3562785501"}]: dispatch 2024-03-26T04:37:14.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[29581]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 189 B/s wr, 295 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:14.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2467035683' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3562785501"}]': finished 2024-03-26T04:37:14.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[29581]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:37:14.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1261520852' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]: dispatch 2024-03-26T04:37:14.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]: dispatch 2024-03-26T04:37:14.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[24814]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 189 B/s wr, 295 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:14.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2467035683' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3562785501"}]': finished 2024-03-26T04:37:14.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[24814]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:37:14.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1261520852' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]: dispatch 2024-03-26T04:37:14.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:14 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]: dispatch 2024-03-26T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:14 smithi129 ceph-mon[25657]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 189 B/s wr, 295 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:14 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2467035683' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3562785501"}]': finished 2024-03-26T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:14 smithi129 ceph-mon[25657]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:14 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1261520852' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]: dispatch 2024-03-26T04:37:14.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:14 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]: dispatch 2024-03-26T04:37:15.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]': finished 2024-03-26T04:37:15.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[24814]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3221116845' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]: dispatch 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]: dispatch 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[24814]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]': finished 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[29581]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3221116845' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]: dispatch 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]: dispatch 2024-03-26T04:37:15.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:15 smithi017 ceph-mon[29581]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-26T04:37:15.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:15 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4071279932"}]': finished 2024-03-26T04:37:15.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:15 smithi129 ceph-mon[25657]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:37:15.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:15 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3221116845' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]: dispatch 2024-03-26T04:37:15.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:15 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]: dispatch 2024-03-26T04:37:15.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:15 smithi129 ceph-mon[25657]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 59 op/s 2024-03-26T04:37:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]': finished 2024-03-26T04:37:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1655690882' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-03-26T04:37:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: Checking dashboard <-> RGW credentials 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]': finished 2024-03-26T04:37:16.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1655690882' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]: dispatch 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]: dispatch 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T04:37:16.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: Checking dashboard <-> RGW credentials 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:16 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4126564882"}]': finished 2024-03-26T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1655690882' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]: dispatch 2024-03-26T04:37:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:16.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:16.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: Checking dashboard <-> RGW credentials 2024-03-26T04:37:16.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:16.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:16 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:37:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]': finished 2024-03-26T04:37:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[24814]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:37:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/504583419' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]: dispatch 2024-03-26T04:37:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]: dispatch 2024-03-26T04:37:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[24814]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-26T04:37:17.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]': finished 2024-03-26T04:37:17.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[29581]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:37:17.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/504583419' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]: dispatch 2024-03-26T04:37:17.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]: dispatch 2024-03-26T04:37:17.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:17 smithi017 ceph-mon[29581]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-26T04:37:17.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:17 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4071279932"}]': finished 2024-03-26T04:37:17.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:17 smithi129 ceph-mon[25657]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:37:17.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:17 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/504583419' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]: dispatch 2024-03-26T04:37:17.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:17 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]: dispatch 2024-03-26T04:37:17.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:17 smithi129 ceph-mon[25657]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-26T04:37:18.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:18] "GET /metrics HTTP/1.1" 200 214989 "" "Prometheus/2.33.4" 2024-03-26T04:37:18.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]': finished 2024-03-26T04:37:18.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-mon[29581]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:37:18.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1820852964' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1317244932"}]: dispatch 2024-03-26T04:37:18.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]': finished 2024-03-26T04:37:18.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-mon[24814]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:37:18.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:18 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1820852964' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1317244932"}]: dispatch 2024-03-26T04:37:18.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:18 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4199384931"}]': finished 2024-03-26T04:37:18.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:18 smithi129 ceph-mon[25657]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:37:18.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:18 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1820852964' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1317244932"}]: dispatch 2024-03-26T04:37:19.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1820852964' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1317244932"}]': finished 2024-03-26T04:37:19.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[24814]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:37:19.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2575970302' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]: dispatch 2024-03-26T04:37:19.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]: dispatch 2024-03-26T04:37:19.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[24814]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-26T04:37:19.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1820852964' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1317244932"}]': finished 2024-03-26T04:37:19.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[29581]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:37:19.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2575970302' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]: dispatch 2024-03-26T04:37:19.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]: dispatch 2024-03-26T04:37:19.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:19 smithi017 ceph-mon[29581]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-26T04:37:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:19 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1820852964' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1317244932"}]': finished 2024-03-26T04:37:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:19 smithi129 ceph-mon[25657]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:37:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:19 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2575970302' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]: dispatch 2024-03-26T04:37:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:19 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]: dispatch 2024-03-26T04:37:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:19 smithi129 ceph-mon[25657]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 255 B/s wr, 14 op/s 2024-03-26T04:37:20.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:37:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:37:20.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]': finished 2024-03-26T04:37:20.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[29581]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:37:20.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3155332587' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]: dispatch 2024-03-26T04:37:20.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]: dispatch 2024-03-26T04:37:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]': finished 2024-03-26T04:37:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[24814]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:37:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3155332587' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]: dispatch 2024-03-26T04:37:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:20 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]: dispatch 2024-03-26T04:37:20.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:20 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4101203109"}]': finished 2024-03-26T04:37:20.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:20 smithi129 ceph-mon[25657]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:37:20.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:20 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3155332587' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]: dispatch 2024-03-26T04:37:20.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:20 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]: dispatch 2024-03-26T04:37:21.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]': finished 2024-03-26T04:37:21.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[24814]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:37:21.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/395580124' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]: dispatch 2024-03-26T04:37:21.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]: dispatch 2024-03-26T04:37:21.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[24814]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:37:21.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]': finished 2024-03-26T04:37:21.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[29581]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:37:21.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/395580124' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]: dispatch 2024-03-26T04:37:21.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]: dispatch 2024-03-26T04:37:21.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:21 smithi017 ceph-mon[29581]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:37:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:21 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/4199384931"}]': finished 2024-03-26T04:37:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:21 smithi129 ceph-mon[25657]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:37:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/395580124' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]: dispatch 2024-03-26T04:37:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:21 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]: dispatch 2024-03-26T04:37:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:21 smithi129 ceph-mon[25657]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:37:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:22 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]': finished 2024-03-26T04:37:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:22 smithi017 ceph-mon[24814]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:37:22.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:22 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4000298258' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4126564882"}]: dispatch 2024-03-26T04:37:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:22 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]': finished 2024-03-26T04:37:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:22 smithi017 ceph-mon[29581]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:37:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:22 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4000298258' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4126564882"}]: dispatch 2024-03-26T04:37:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:22 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2717172247"}]': finished 2024-03-26T04:37:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:22 smithi129 ceph-mon[25657]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:37:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:22 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4000298258' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4126564882"}]: dispatch 2024-03-26T04:37:23.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4000298258' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4126564882"}]': finished 2024-03-26T04:37:23.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[29581]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:37:23.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:23.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1409238386' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]: dispatch 2024-03-26T04:37:23.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]: dispatch 2024-03-26T04:37:23.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[29581]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:23.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:37:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:37:23.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:37:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:37:23.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:37:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:37:23.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4000298258' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4126564882"}]': finished 2024-03-26T04:37:23.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[24814]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:37:23.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:23.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1409238386' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]: dispatch 2024-03-26T04:37:23.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]: dispatch 2024-03-26T04:37:23.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:23 smithi017 ceph-mon[24814]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:23 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4000298258' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/4126564882"}]': finished 2024-03-26T04:37:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:23 smithi129 ceph-mon[25657]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:37:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:23 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:23 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1409238386' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]: dispatch 2024-03-26T04:37:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:23 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]: dispatch 2024-03-26T04:37:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:23 smithi129 ceph-mon[25657]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:24 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]': finished 2024-03-26T04:37:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:24 smithi129 ceph-mon[25657]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:37:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:24 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3389764497' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]: dispatch 2024-03-26T04:37:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:24 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]: dispatch 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]': finished 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[29581]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3389764497' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]: dispatch 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]: dispatch 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/27415178"}]': finished 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[24814]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:37:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3389764497' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]: dispatch 2024-03-26T04:37:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:24 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]: dispatch 2024-03-26T04:37:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:25 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]': finished 2024-03-26T04:37:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:25 smithi129 ceph-mon[25657]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:37:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:25 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4171665893' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]: dispatch 2024-03-26T04:37:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:25 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]: dispatch 2024-03-26T04:37:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:25 smithi129 ceph-mon[25657]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:25.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]': finished 2024-03-26T04:37:25.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[29581]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:37:25.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4171665893' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]: dispatch 2024-03-26T04:37:25.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]: dispatch 2024-03-26T04:37:25.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[29581]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:25.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1886501174"}]': finished 2024-03-26T04:37:25.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[24814]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:37:25.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4171665893' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]: dispatch 2024-03-26T04:37:25.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]: dispatch 2024-03-26T04:37:25.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:25 smithi017 ceph-mon[24814]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:26.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:26 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]': finished 2024-03-26T04:37:26.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:26 smithi129 ceph-mon[25657]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:37:26.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:26 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/174190302' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1272584175"}]: dispatch 2024-03-26T04:37:26.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:26 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]': finished 2024-03-26T04:37:26.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:26 smithi017 ceph-mon[29581]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:37:26.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:26 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/174190302' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1272584175"}]: dispatch 2024-03-26T04:37:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:26 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3729939084"}]': finished 2024-03-26T04:37:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:26 smithi017 ceph-mon[24814]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:37:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:26 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/174190302' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1272584175"}]: dispatch 2024-03-26T04:37:27.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:27 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/174190302' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1272584175"}]': finished 2024-03-26T04:37:27.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:27 smithi129 ceph-mon[25657]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:37:27.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:27 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1603072154' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]: dispatch 2024-03-26T04:37:27.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:27 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]: dispatch 2024-03-26T04:37:27.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:27 smithi129 ceph-mon[25657]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:27.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/174190302' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1272584175"}]': finished 2024-03-26T04:37:27.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[24814]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:37:27.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1603072154' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]: dispatch 2024-03-26T04:37:27.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]: dispatch 2024-03-26T04:37:27.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[24814]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:27.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/174190302' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1272584175"}]': finished 2024-03-26T04:37:27.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[29581]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:37:27.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1603072154' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]: dispatch 2024-03-26T04:37:27.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]: dispatch 2024-03-26T04:37:27.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:27 smithi017 ceph-mon[29581]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:28.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:28 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]': finished 2024-03-26T04:37:28.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:28 smithi129 ceph-mon[25657]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:37:28.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:28 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2484501516' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]: dispatch 2024-03-26T04:37:28.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:28 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]: dispatch 2024-03-26T04:37:28.956 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:28] "GET /metrics HTTP/1.1" 200 214989 "" "Prometheus/2.33.4" 2024-03-26T04:37:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]': finished 2024-03-26T04:37:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[24814]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:37:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2484501516' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]: dispatch 2024-03-26T04:37:28.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]: dispatch 2024-03-26T04:37:28.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2000177021"}]': finished 2024-03-26T04:37:28.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[29581]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:37:28.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2484501516' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]: dispatch 2024-03-26T04:37:28.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:28 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]: dispatch 2024-03-26T04:37:29.809 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:29 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]': finished 2024-03-26T04:37:29.809 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:29 smithi129 ceph-mon[25657]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:37:29.809 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:29 smithi129 ceph-mon[25657]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:29 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]': finished 2024-03-26T04:37:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:29 smithi017 ceph-mon[24814]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:37:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:29 smithi017 ceph-mon[24814]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:29.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:29 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1619556253"}]': finished 2024-03-26T04:37:29.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:29 smithi017 ceph-mon[29581]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:37:29.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:29 smithi017 ceph-mon[29581]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:30.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:37:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:37:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:32 smithi129 ceph-mon[25657]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-26T04:37:32.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:32 smithi017 ceph-mon[24814]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-26T04:37:32.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:32 smithi017 ceph-mon[29581]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-26T04:37:33.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:37:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:37:33.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:37:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:37:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:37:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:37:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:34 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:34 smithi129 ceph-mon[25657]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:37:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:34 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:34 smithi017 ceph-mon[29581]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:37:34.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:34 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:34.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:34 smithi017 ceph-mon[24814]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:37:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:36 smithi129 ceph-mon[25657]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:37:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:36 smithi017 ceph-mon[24814]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:37:36.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:36 smithi017 ceph-mon[29581]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:37:38.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:38 smithi129 ceph-mon[25657]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:37:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:38 smithi017 ceph-mon[24814]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:37:38.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:38 smithi017 ceph-mon[29581]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:37:38.956 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:37:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:38] "GET /metrics HTTP/1.1" 200 215005 "" "Prometheus/2.33.4" 2024-03-26T04:37:40.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:37:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:37:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:40 smithi129 ceph-mon[25657]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-03-26T04:37:40.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:40 smithi017 ceph-mon[24814]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-03-26T04:37:40.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:40 smithi017 ceph-mon[29581]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-03-26T04:37:42.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:42 smithi017 ceph-mon[24814]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:42.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:42 smithi017 ceph-mon[29581]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:42 smithi129 ceph-mon[25657]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:43.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:37:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:37:43.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:37:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:44 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:44 smithi129 ceph-mon[25657]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:44 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:44 smithi017 ceph-mon[24814]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:44.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:44 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:44.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:44 smithi017 ceph-mon[29581]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:46 smithi129 ceph-mon[25657]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:46.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:46 smithi017 ceph-mon[24814]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:46.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:46 smithi017 ceph-mon[29581]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:48 smithi129 ceph-mon[25657]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:48 smithi017 ceph-mon[24814]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:48 smithi017 ceph-mon[29581]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:48.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:37:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:48] "GET /metrics HTTP/1.1" 200 214957 "" "Prometheus/2.33.4" 2024-03-26T04:37:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:50 smithi129 ceph-mon[25657]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:50.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:37:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:37:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:50 smithi017 ceph-mon[29581]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:50.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:50 smithi017 ceph-mon[24814]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:52 smithi129 ceph-mon[25657]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:52 smithi017 ceph-mon[24814]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:52 smithi017 ceph-mon[29581]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:54 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:54 smithi129 ceph-mon[25657]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:54 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:54 smithi017 ceph-mon[24814]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:54 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:54.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:54 smithi017 ceph-mon[29581]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:56 smithi129 ceph-mon[25657]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:56 smithi017 ceph-mon[24814]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:56.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:56 smithi017 ceph-mon[29581]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:58.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:37:58 smithi129 ceph-mon[25657]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:58.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:37:58 smithi017 ceph-mon[24814]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:58.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:37:58 smithi017 ceph-mon[29581]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:58.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:37:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:58] "GET /metrics HTTP/1.1" 200 214957 "" "Prometheus/2.33.4" 2024-03-26T04:38:00.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:37:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:37:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:38:00.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:00 smithi129 ceph-mon[25657]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:00 smithi017 ceph-mon[24814]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:00.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:00 smithi017 ceph-mon[29581]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:38:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:38:01.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:38:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:02 smithi129 ceph-mon[25657]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:02 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:38:02.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:02 smithi017 ceph-mon[24814]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:02.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:02 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:38:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:02 smithi017 ceph-mon[29581]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:02 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:38:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:04 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:04.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:04 smithi129 ceph-mon[25657]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:04 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:04.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:04 smithi017 ceph-mon[24814]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:04.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:04 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:04.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:04 smithi017 ceph-mon[29581]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:06 smithi129 ceph-mon[25657]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:06 smithi017 ceph-mon[24814]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:06.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:06 smithi017 ceph-mon[29581]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:08 smithi129 ceph-mon[25657]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:08 smithi017 ceph-mon[24814]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:08 smithi017 ceph-mon[29581]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:08.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:38:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:08] "GET /metrics HTTP/1.1" 200 214913 "" "Prometheus/2.33.4" 2024-03-26T04:38:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:10 smithi129 ceph-mon[25657]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:10.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:38:09 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:38:10.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:10 smithi017 ceph-mon[24814]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:10.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:10 smithi017 ceph-mon[29581]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:12 smithi129 ceph-mon[25657]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:12.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:12 smithi017 ceph-mon[24814]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:12.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:12 smithi017 ceph-mon[29581]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:14 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:14 smithi129 ceph-mon[25657]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:14 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:14 smithi017 ceph-mon[29581]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:14 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:14 smithi017 ceph-mon[24814]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:16 smithi129 ceph-mon[25657]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:16.423 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:16 smithi017 ceph-mon[29581]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:16.423 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:16 smithi017 ceph-mon[24814]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:17 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:17 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:17 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:38:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:17 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:17 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:17 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:38:17.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:17 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:17.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:17 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:17.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:17 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:38:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:18 smithi129 ceph-mon[25657]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:18 smithi017 ceph-mon[24814]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:18.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:18 smithi017 ceph-mon[29581]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:18.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:38:18 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:18] "GET /metrics HTTP/1.1" 200 214915 "" "Prometheus/2.33.4" 2024-03-26T04:38:20.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:38:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:38:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:20 smithi129 ceph-mon[25657]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:20 smithi017 ceph-mon[24814]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:20 smithi017 ceph-mon[29581]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:22 smithi129 ceph-mon[25657]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:22 smithi017 ceph-mon[24814]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:22 smithi017 ceph-mon[29581]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:24 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:24 smithi129 ceph-mon[25657]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:24 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:24 smithi017 ceph-mon[24814]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:24 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:24 smithi017 ceph-mon[29581]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:26 smithi129 ceph-mon[25657]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:26 smithi017 ceph-mon[24814]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:26.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:26 smithi017 ceph-mon[29581]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:28 smithi129 ceph-mon[25657]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:28 smithi017 ceph-mon[24814]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:28.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:28 smithi017 ceph-mon[29581]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:28.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:38:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:28] "GET /metrics HTTP/1.1" 200 214915 "" "Prometheus/2.33.4" 2024-03-26T04:38:30.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:38:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:38:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:30 smithi129 ceph-mon[25657]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:30.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:30 smithi017 ceph-mon[29581]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:30.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:30 smithi017 ceph-mon[24814]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:32 smithi129 ceph-mon[25657]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:32.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:32 smithi017 ceph-mon[24814]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:32.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:32 smithi017 ceph-mon[29581]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:34 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:34 smithi129 ceph-mon[25657]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:34 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:34 smithi017 ceph-mon[24814]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:34 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:34.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:34 smithi017 ceph-mon[29581]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:36 smithi129 ceph-mon[25657]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:36 smithi017 ceph-mon[24814]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:36.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:36 smithi017 ceph-mon[29581]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:38.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:38 smithi129 ceph-mon[25657]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:38 smithi017 ceph-mon[24814]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:38.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:38 smithi017 ceph-mon[29581]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:38.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:38:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:38] "GET /metrics HTTP/1.1" 200 214908 "" "Prometheus/2.33.4" 2024-03-26T04:38:40.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:38:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:38:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:40 smithi129 ceph-mon[25657]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:40.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:40 smithi017 ceph-mon[24814]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:40.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:40 smithi017 ceph-mon[29581]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:42 smithi129 ceph-mon[25657]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:42.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:42 smithi017 ceph-mon[24814]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:42.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:42 smithi017 ceph-mon[29581]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:44 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:44 smithi129 ceph-mon[25657]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:44 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:44 smithi017 ceph-mon[24814]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:44.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:44 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:44.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:44 smithi017 ceph-mon[29581]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:46 smithi129 ceph-mon[25657]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:46.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:46 smithi017 ceph-mon[24814]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:46.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:46 smithi017 ceph-mon[29581]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:48 smithi129 ceph-mon[25657]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:48 smithi017 ceph-mon[24814]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:48.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:48 smithi017 ceph-mon[29581]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:48.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:38:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:48] "GET /metrics HTTP/1.1" 200 214910 "" "Prometheus/2.33.4" 2024-03-26T04:38:50.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:38:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:38:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:50 smithi129 ceph-mon[25657]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:50.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:50 smithi017 ceph-mon[24814]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:50 smithi017 ceph-mon[29581]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:52 smithi129 ceph-mon[25657]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:52 smithi017 ceph-mon[24814]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:52 smithi017 ceph-mon[29581]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:54 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:54 smithi129 ceph-mon[25657]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:54 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:54 smithi017 ceph-mon[24814]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:54 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:54 smithi017 ceph-mon[29581]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:56 smithi129 ceph-mon[25657]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:56 smithi017 ceph-mon[24814]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:56.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:56 smithi017 ceph-mon[29581]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:58.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:38:58 smithi129 ceph-mon[25657]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:58.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:38:58 smithi017 ceph-mon[24814]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:58.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:38:58 smithi017 ceph-mon[29581]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:58.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:38:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:58] "GET /metrics HTTP/1.1" 200 214910 "" "Prometheus/2.33.4" 2024-03-26T04:39:00.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:00 smithi129 ceph-mon[25657]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:00.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:38:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:38:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:39:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:00 smithi017 ceph-mon[24814]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:00 smithi017 ceph-mon[29581]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:39:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:39:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:39:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:39:01.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:39:01.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:39:02.385 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:02 smithi017 ceph-mon[24814]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:02.385 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:02 smithi017 ceph-mon[29581]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:02 smithi129 ceph-mon[25657]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:03.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-26T04:39:04.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:04 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:04.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:04 smithi017 ceph-mon[24814]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:04.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:04 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:04.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:04 smithi017 ceph-mon[29581]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:04 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:04 smithi129 ceph-mon[25657]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:05.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-26T04:39:06.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:06 smithi017 ceph-mon[24814]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:06.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:06 smithi017 ceph-mon[29581]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:06 smithi129 ceph-mon[25657]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:08 smithi017 ceph-mon[24814]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:08.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:08 smithi017 ceph-mon[29581]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:08.205 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-26T04:39:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:08 smithi129 ceph-mon[25657]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:08.752 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:39:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:08] "GET /metrics HTTP/1.1" 200 214910 "" "Prometheus/2.33.4" 2024-03-26T04:39:10.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:10 smithi017 ceph-mon[24814]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:10.355 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:10 smithi017 ceph-mon[29581]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:10 smithi129 ceph-mon[25657]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:10.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:39:09 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:39:10.596 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-26T04:39:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:12 smithi129 ceph-mon[25657]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:12.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:12 smithi017 ceph-mon[24814]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:12.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:12 smithi017 ceph-mon[29581]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:12.541 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.171 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T04:39:13.181 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-03-26T04:39:13.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-26T04:39:13.833 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[29581]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.834 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:13.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:13.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:13.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:13.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:13.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:13 smithi017 ceph-mon[24814]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:13.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:13 smithi129 ceph-mon[25657]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:15.441 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:39:16.190 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:39:16.190 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 2m ago 3m 15.4M - ba2b418f427c f2a5289b41b6 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (2m) 2m ago 2m 48.3M - 8.3.5 dad864ee21e9 47e825a337c6 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (2m) 2m ago 2m 69.5M - 3.5 e1d6a67b021e a7492721d53e 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443 running (7m) 2m ago 7m 410M - 17.2.0 e1d6a67b021e c287d82f4db1 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 9b468d7472cd 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (9m) 2m ago 9m 47.5M 2048M 17.2.0 e1d6a67b021e e512d90ccc03 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (7m) 2m ago 7m 38.3M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (7m) 2m ago 7m 38.0M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 2m ago 3m 15.3M - 1dbe0e931976 af8933711de0 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (3m) 2m ago 3m 16.3M - 1dbe0e931976 d147c628643f 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (7m) 2m ago 7m 50.2M 4017M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (6m) 2m ago 6m 52.6M 4017M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (6m) 2m ago 6m 48.2M 4017M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (6m) 2m ago 6m 50.2M 4017M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:39:16.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (5m) 2m ago 5m 50.8M 4272M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (5m) 2m ago 5m 47.2M 4272M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (4m) 2m ago 4m 51.0M 4272M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (4m) 2m ago 4m 49.0M 4272M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (2m) 2m ago 3m 43.9M - 514e6a882f6e faf1d87a92d7 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (2m) 2m ago 2m 69.1M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (2m) 2m ago 2m 71.8M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (2m) 2m ago 2m 69.5M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:39:16.192 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (2m) 2m ago 2m 69.0M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:39:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:16 smithi129 ceph-mon[25657]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:16.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:16 smithi017 ceph-mon[24814]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:16.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:16 smithi017 ceph-mon[29581]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:39:16.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:39:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-26T04:39:16.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:39:16.640 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "", 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a image" 2024-03-26T04:39:17.041 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:39:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:17 smithi129 ceph-mon[25657]: from='client.24937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:17 smithi129 ceph-mon[25657]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:17 smithi129 ceph-mon[25657]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:17 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1780371771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[29581]: from='client.24937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[29581]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[29581]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1780371771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[24814]: from='client.24937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[24814]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[24814]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:17.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:17 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1780371771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:39:17.539 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:39:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:18 smithi129 ceph-mon[25657]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:18 smithi129 ceph-mon[25657]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:18 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4094406466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:39:18.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-mon[29581]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:18.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-mon[29581]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:18.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4094406466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:39:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-mon[24814]: from='client.24961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-mon[24814]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4094406466' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:39:18.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:39:18 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:18] "GET /metrics HTTP/1.1" 200 214910 "" "Prometheus/2.33.4" 2024-03-26T04:39:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:20 smithi129 ceph-mon[25657]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:20.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:39:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:39:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:20 smithi017 ceph-mon[24814]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:20.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:20 smithi017 ceph-mon[29581]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:22 smithi129 ceph-mon[25657]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:22 smithi017 ceph-mon[24814]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:22 smithi017 ceph-mon[29581]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:24 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:24 smithi129 ceph-mon[25657]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:24 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:24 smithi017 ceph-mon[29581]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:24 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:24 smithi017 ceph-mon[24814]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:26 smithi129 ceph-mon[25657]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:26 smithi017 ceph-mon[24814]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:26.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:26 smithi017 ceph-mon[29581]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:28 smithi129 ceph-mon[25657]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:28 smithi017 ceph-mon[24814]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:28.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:28 smithi017 ceph-mon[29581]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:28.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:39:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:28] "GET /metrics HTTP/1.1" 200 214910 "" "Prometheus/2.33.4" 2024-03-26T04:39:30.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:30 smithi129 ceph-mon[25657]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:30.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:39:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:39:30.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:30 smithi017 ceph-mon[24814]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:30.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:30 smithi017 ceph-mon[29581]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:32 smithi129 ceph-mon[25657]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:32.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:32 smithi017 ceph-mon[24814]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:32.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:32 smithi017 ceph-mon[29581]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:34 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:34 smithi129 ceph-mon[25657]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:34 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:34 smithi017 ceph-mon[24814]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:34 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:34 smithi017 ceph-mon[29581]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:36.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:36 smithi129 ceph-mon[25657]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:36.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:36 smithi017 ceph-mon[24814]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:36.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:36 smithi017 ceph-mon[29581]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:38.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:38 smithi129 ceph-mon[25657]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:38.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:38 smithi017 ceph-mon[24814]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:38.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:38 smithi017 ceph-mon[29581]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:38.926 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:39:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:38] "GET /metrics HTTP/1.1" 200 214893 "" "Prometheus/2.33.4" 2024-03-26T04:39:40.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[29581]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:40.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:40.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:40.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:40.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[24814]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:40.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:40.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:40.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:40 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:40.244 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:39:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:39:40.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:40 smithi129 ceph-mon[25657]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:40.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:40 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:40.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:40 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:40.245 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:40 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:39:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:41 smithi129 ceph-mon[25657]: Upgrade: Target is version 19.0.0-1775-g5e70124b (unknown) 2024-03-26T04:39:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:41 smithi129 ceph-mon[25657]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2'] 2024-03-26T04:39:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:41 smithi129 ceph-mon[25657]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-26T04:39:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:41 smithi129 ceph-mon[25657]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on smithi129 2024-03-26T04:39:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[24814]: Upgrade: Target is version 19.0.0-1775-g5e70124b (unknown) 2024-03-26T04:39:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[24814]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2'] 2024-03-26T04:39:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[24814]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-26T04:39:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[24814]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on smithi129 2024-03-26T04:39:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[29581]: Upgrade: Target is version 19.0.0-1775-g5e70124b (unknown) 2024-03-26T04:39:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[29581]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2'] 2024-03-26T04:39:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[29581]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-26T04:39:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:41 smithi017 ceph-mon[29581]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on smithi129 2024-03-26T04:39:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:42 smithi129 ceph-mon[25657]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:42.403 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:42 smithi017 ceph-mon[24814]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:42.404 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:42 smithi017 ceph-mon[29581]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:44 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:44.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:44 smithi129 ceph-mon[25657]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:44 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:44.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:44 smithi017 ceph-mon[24814]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:44.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:44 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:44.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:44 smithi017 ceph-mon[29581]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:46 smithi129 ceph-mon[25657]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:46.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:46 smithi017 ceph-mon[24814]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:46.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:46 smithi017 ceph-mon[29581]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:47.873 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:39:48.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:48 smithi129 ceph-mon[25657]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:48 smithi017 ceph-mon[24814]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:48 smithi017 ceph-mon[29581]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:48.477 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:39:48.477 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 2m ago 3m 15.4M - ba2b418f427c f2a5289b41b6 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (3m) 2m ago 3m 48.3M - 8.3.5 dad864ee21e9 47e825a337c6 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (2m) 2m ago 2m 69.5M - 3.5 e1d6a67b021e a7492721d53e 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443 running (8m) 2m ago 8m 410M - 17.2.0 e1d6a67b021e c287d82f4db1 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e 9b468d7472cd 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (9m) 2m ago 9m 47.5M 2048M 17.2.0 e1d6a67b021e e512d90ccc03 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (8m) 2m ago 8m 38.3M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (8m) 2m ago 8m 38.0M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 2m ago 4m 15.3M - 1dbe0e931976 af8933711de0 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (4m) 2m ago 4m 16.3M - 1dbe0e931976 d147c628643f 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (7m) 2m ago 7m 50.2M 4017M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (7m) 2m ago 7m 52.6M 4017M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (7m) 2m ago 7m 48.2M 4017M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (6m) 2m ago 6m 50.2M 4017M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:39:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (6m) 2m ago 6m 50.8M 4272M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (5m) 2m ago 5m 47.2M 4272M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (5m) 2m ago 5m 51.0M 4272M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (5m) 2m ago 5m 49.0M 4272M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (3m) 2m ago 4m 43.9M - 514e6a882f6e faf1d87a92d7 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (2m) 2m ago 2m 69.1M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (2m) 2m ago 2m 71.8M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (2m) 2m ago 2m 69.5M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:39:48.479 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (2m) 2m ago 2m 69.0M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:39:48.842 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:39:48.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:39:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:39:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-26T04:39:48.844 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:39:48.844 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:39:48.844 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:39:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:48] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "0/23 daemons upgraded", 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a image on host smithi129" 2024-03-26T04:39:49.150 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:39:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:49 smithi129 ceph-mon[25657]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:49 smithi129 ceph-mon[25657]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:49.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:49 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3323880215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:39:49.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:49 smithi017 ceph-mon[29581]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:49.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:49 smithi017 ceph-mon[29581]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:49.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:49 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3323880215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:39:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:49 smithi017 ceph-mon[24814]: from='client.24973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:49 smithi017 ceph-mon[24814]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:49 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3323880215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:39:49.514 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:39:50.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:39:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:39:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:50 smithi129 ceph-mon[25657]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:50 smithi129 ceph-mon[25657]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:50 smithi129 ceph-mon[25657]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:50 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1871571117' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:39:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[29581]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[29581]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[29581]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1871571117' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:39:50.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[24814]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:50.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[24814]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:50.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[24814]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:39:50.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:50 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1871571117' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:39:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:52 smithi129 ceph-mon[25657]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:52 smithi017 ceph-mon[24814]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:52 smithi017 ceph-mon[29581]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:53.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:39:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:39:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:39:53.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:39:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:39:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:39:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:54 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:54.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:54 smithi129 ceph-mon[25657]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:54 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:54 smithi017 ceph-mon[24814]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:54 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:54 smithi017 ceph-mon[29581]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:56 smithi129 ceph-mon[25657]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:56 smithi017 ceph-mon[24814]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:56.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:56 smithi017 ceph-mon[29581]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:58.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:39:58 smithi129 ceph-mon[25657]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:58.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:39:58 smithi017 ceph-mon[24814]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:58.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:39:58 smithi017 ceph-mon[29581]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:58.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:39:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:58] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-03-26T04:40:00.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:39:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:39:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:40:00.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:00 smithi129 ceph-mon[25657]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:00.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:00 smithi129 ceph-mon[25657]: overall HEALTH_OK 2024-03-26T04:40:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:00 smithi017 ceph-mon[24814]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:00 smithi017 ceph-mon[24814]: overall HEALTH_OK 2024-03-26T04:40:00.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:00 smithi017 ceph-mon[29581]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:00.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:00 smithi017 ceph-mon[29581]: overall HEALTH_OK 2024-03-26T04:40:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:01.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:01 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:01.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:01.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:01 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:01.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:01 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:02.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:02 smithi129 ceph-mon[25657]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:02.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:02 smithi017 ceph-mon[24814]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:02 smithi017 ceph-mon[29581]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:03.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:40:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:03.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:03.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:04 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:04 smithi129 ceph-mon[25657]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:04.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:04 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:04.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:04 smithi017 ceph-mon[29581]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:04.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:04 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:04.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:04 smithi017 ceph-mon[24814]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:06 smithi129 ceph-mon[25657]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:06.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:06 smithi017 ceph-mon[24814]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:06.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:06 smithi017 ceph-mon[29581]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:08.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:08 smithi129 ceph-mon[25657]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:08.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:08 smithi017 ceph-mon[24814]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:08.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:08 smithi017 ceph-mon[29581]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:08.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:08] "GET /metrics HTTP/1.1" 200 214858 "" "Prometheus/2.33.4" 2024-03-26T04:40:10.391 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:10 smithi129 ceph-mon[25657]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:10.392 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:09 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[26852]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:40:10.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:10 smithi017 ceph-mon[29581]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:10.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:10 smithi017 ceph-mon[24814]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:12.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:12 smithi129 ceph-mon[25657]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:12.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:12 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:12.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:12 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:40:12.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:12 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:40:12.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:12 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:12.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[24814]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:12.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:12.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:40:12.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:40:12.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:12.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[29581]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:12.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:12.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:40:12.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:40:12.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:12 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:13.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:13 smithi129 ceph-mon[25657]: Upgrade: Updating mgr.x 2024-03-26T04:40:13.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:13 smithi129 ceph-mon[25657]: Deploying daemon mgr.x on smithi129 2024-03-26T04:40:13.363 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:13 smithi129 systemd[1]: Stopping Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:40:13.363 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:13 smithi129 bash[46621]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr.x" found: no such container 2024-03-26T04:40:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-mon[24814]: Upgrade: Updating mgr.x 2024-03-26T04:40:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-mon[24814]: Deploying daemon mgr.x on smithi129 2024-03-26T04:40:13.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-mon[29581]: Upgrade: Updating mgr.x 2024-03-26T04:40:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-mon[29581]: Deploying daemon mgr.x on smithi129 2024-03-26T04:40:13.627 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:13 smithi129 podman[46632]: 2024-03-26 04:40:13.362799171 +0000 UTC m=+0.169671071 container died c287d82f4db12707d9ce636247d7deefbb05a092bc03785c5eb8275d9bc58eb2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, version=8, distribution-scope=public, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-03-26T04:40:13.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:13 smithi129 podman[46632]: 2024-03-26 04:40:13.792286307 +0000 UTC m=+0.599158205 container cleanup c287d82f4db12707d9ce636247d7deefbb05a092bc03785c5eb8275d9bc58eb2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, version=8) 2024-03-26T04:40:13.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:13 smithi129 bash[46632]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:40:13.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:13 smithi129 bash[46657]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr.x" found: no such container 2024-03-26T04:40:13.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:40:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:13.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:40:13.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:13 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:14 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:14 smithi129 ceph-mon[25657]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:14.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 podman[46646]: 2024-03-26 04:40:14.010760889 +0000 UTC m=+0.644485632 container remove c287d82f4db12707d9ce636247d7deefbb05a092bc03785c5eb8275d9bc58eb2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, name=centos-stream, ceph=True, architecture=x86_64, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, release=754, vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T04:40:14.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:40:14.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-26T04:40:14.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 systemd[1]: Stopped Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:40:14.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service: Consumed 16.465s CPU time. 2024-03-26T04:40:14.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 systemd[1]: Starting Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:40:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:14 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:14 smithi017 ceph-mon[24814]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:14.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:14 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:14.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:14 smithi017 ceph-mon[29581]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:14.722 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 podman[46767]: 2024-03-26 04:40:14.330095465 +0000 UTC m=+0.018151930 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:40:14.722 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 podman[46767]: 2024-03-26 04:40:14.436909186 +0000 UTC m=+0.124965637 container create 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T04:40:14.722 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 podman[46767]: 2024-03-26 04:40:14.63928759 +0000 UTC m=+0.327344042 container init 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:40:14.722 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 podman[46767]: 2024-03-26 04:40:14.642216077 +0000 UTC m=+0.330272539 container start 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:40:14.722 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 ceph-mgr[46787]: -- 172.21.15.129:0/3471683345 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8270fe1a0 con 0x55c8270d5000 2024-03-26T04:40:15.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 bash[46767]: 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 2024-03-26T04:40:15.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:14 smithi129 systemd[1]: Started Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:40:15.507 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:15.230+0000 7faef58a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:40:15.507 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:15.323+0000 7faef58a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:40:16.108 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.013+0000 7faef58a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:40:16.108 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.107+0000 7faef58a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:40:16.108 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:15 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:16.108 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:15 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:16.108 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:15 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:16.108 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:15 smithi129 ceph-mon[25657]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:16.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:16.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:16.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:16.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[24814]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:16.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:16.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:16.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:16.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:15 smithi017 ceph-mon[29581]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:16.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.256+0000 7faef58a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:40:16.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.356+0000 7faef58a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:40:16.741 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.440+0000 7faef58a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:40:16.741 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.740+0000 7faef58a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:40:17.045 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.854+0000 7faef58a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:17.045 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:16.944+0000 7faef58a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:40:17.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:17.044+0000 7faef58a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:40:17.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:17.232+0000 7faef58a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:40:18.382 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:18.244+0000 7faef58a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:40:18.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:18 smithi129 ceph-mon[25657]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:18 smithi017 ceph-mon[24814]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:18.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:18 smithi017 ceph-mon[29581]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:18.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:18.376+0000 7faef58a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:40:18.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:18 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:18] "GET /metrics HTTP/1.1" 200 214847 "" "Prometheus/2.33.4" 2024-03-26T04:40:19.016 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:18.726+0000 7faef58a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:40:19.016 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:18.817+0000 7faef58a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:40:19.016 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:19.016+0000 7faef58a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:19.204+0000 7faef58a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:40:19.771 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:19.490+0000 7faef58a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:19.848 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:40:20.086 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:19.752+0000 7faef58a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:20 smithi129 ceph-mon[25657]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:20.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:19.991+0000 7faef58a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:40:20.453 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:40:20.453 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 3m ago 4m 15.4M - ba2b418f427c f2a5289b41b6 2024-03-26T04:40:20.453 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (3m) 3m ago 3m 48.3M - 8.3.5 dad864ee21e9 47e825a337c6 2024-03-26T04:40:20.453 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (3m) 3m ago 3m 69.5M - 3.5 e1d6a67b021e a7492721d53e 2024-03-26T04:40:20.453 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283 starting - - - - 2024-03-26T04:40:20.453 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e 9b468d7472cd 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (10m) 3m ago 10m 47.5M 2048M 17.2.0 e1d6a67b021e e512d90ccc03 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (8m) 3m ago 8m 38.3M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (9m) 3m ago 9m 38.0M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 3m ago 4m 15.3M - 1dbe0e931976 af8933711de0 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (4m) 3m ago 4m 16.3M - 1dbe0e931976 d147c628643f 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (8m) 3m ago 8m 50.2M 4017M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (7m) 3m ago 7m 52.6M 4017M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (7m) 3m ago 7m 48.2M 4017M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (7m) 3m ago 7m 50.2M 4017M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (6m) 3m ago 6m 50.8M 4272M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (6m) 3m ago 6m 47.2M 4272M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (6m) 3m ago 6m 51.0M 4272M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (5m) 3m ago 5m 49.0M 4272M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:40:20.454 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (3m) 3m ago 5m 43.9M - 514e6a882f6e faf1d87a92d7 2024-03-26T04:40:20.455 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (3m) 3m ago 3m 69.1M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:40:20.455 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (3m) 3m ago 3m 71.8M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:40:20.455 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (3m) 3m ago 3m 69.5M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:40:20.455 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (3m) 3m ago 3m 69.0M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:40:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:20 smithi017 ceph-mon[24814]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:20.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:20 smithi017 ceph-mon[29581]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:40:20.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:40:20.816 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:40:21.121 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:40:21.121 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:40:21.122 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:40:21.122 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-03-26T04:40:21.122 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "", 2024-03-26T04:40:21.122 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mgr daemons" 2024-03-26T04:40:21.122 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:40:21.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:20.799+0000 7faef58a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:40:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:21 smithi017 ceph-mon[24814]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:21 smithi017 ceph-mon[24814]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/660707340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:40:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:21 smithi017 ceph-mon[29581]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:21 smithi017 ceph-mon[29581]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/660707340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:40:21.485 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:40:21.486 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:21.384+0000 7faef58a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:40:21.487 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-mon[25657]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:21.487 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-mon[25657]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:21.487 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/660707340' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:40:21.824 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:21.486+0000 7faef58a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:40:21.824 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:21.705+0000 7faef58a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:40:21.824 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:21] ENGINE Bus STARTING 2024-03-26T04:40:22.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: CherryPy Checker: 2024-03-26T04:40:22.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: The Application mounted at '' has an empty config. 2024-03-26T04:40:22.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:40:22.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:21] ENGINE Serving on http://:::9283 2024-03-26T04:40:22.146 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:21 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:21] ENGINE Bus STARTED 2024-03-26T04:40:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1975107137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: Standby manager daemon x restarted 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: Standby manager daemon x started 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1975107137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:40:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: Standby manager daemon x restarted 2024-03-26T04:40:22.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: Standby manager daemon x started 2024-03-26T04:40:22.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:40:22.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:40:22.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:40:22.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:22 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1975107137' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: Standby manager daemon x restarted 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: Standby manager daemon x started 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:40:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:40:22.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:40:22.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:22 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:40:23.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:23 smithi017 ceph-mon[29581]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-26T04:40:23.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:23 smithi017 ceph-mon[24814]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-26T04:40:23.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:23 smithi129 ceph-mon[25657]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-26T04:40:23.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:40:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:40:23.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:23.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:23 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:24 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:24 smithi017 ceph-mon[24814]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:24 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:24.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:24 smithi017 ceph-mon[29581]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:24.472 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:24 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:24.473 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:24 smithi129 ceph-mon[25657]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:26 smithi129 ceph-mon[25657]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:26 smithi017 ceph-mon[24814]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:26.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:26 smithi017 ceph-mon[29581]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:28 smithi129 ceph-mon[25657]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:28 smithi017 ceph-mon[24814]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:28 smithi017 ceph-mon[29581]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:28.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:28] "GET /metrics HTTP/1.1" 200 214847 "" "Prometheus/2.33.4" 2024-03-26T04:40:30.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:40:30.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:30 smithi129 ceph-mon[25657]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:30.408 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:30 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:30.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:30 smithi017 ceph-mon[24814]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:30.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:30 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:30.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:30 smithi017 ceph-mon[29581]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:30.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:30 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:31.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:31 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:31.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:31 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:31.760 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:31 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:32.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:32 smithi017 ceph-mon[24814]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:32.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:32 smithi017 ceph-mon[29581]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:32.784 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:32 smithi129 ceph-mon[25657]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:33.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:40:33.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:33.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:34.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:34.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:34.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[29581]: Standby manager daemon y started 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:34.548+0000 7f12632bf700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setuser ceph since I am not root 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ignoring --setgroup ceph since I am not root 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mgr[25046]: -- 172.21.15.17:0/812292135 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f42384340 con 0x558f4235d000 2024-03-26T04:40:34.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:40:34.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-mon[24814]: Standby manager daemon y started 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:34] ENGINE Bus STOPPING 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:34] ENGINE Bus STOPPED 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:34] ENGINE Bus STARTING 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:40:34.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-mon[25657]: Standby manager daemon y started 2024-03-26T04:40:35.074 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:34.799Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:40:35.074 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:34.829Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:40:35.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:34] ENGINE Serving on http://:::9283 2024-03-26T04:40:35.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:40:34] ENGINE Bus STARTED 2024-03-26T04:40:35.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:35.074+0000 7f0860d12000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:40:35.426 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:35.170+0000 7f0860d12000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:40:35.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: Failing over to other MGR 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: mgrmap e22: x(active, starting, since 0.999697s), standbys: y 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:40:35.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: Manager daemon x is now available 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: Queued rgw.foo for migration 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: Queued rgw.smpl for migration 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: Failing over to other MGR 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-26T04:40:35.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: mgrmap e22: x(active, starting, since 0.999697s), standbys: y 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:40:35.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: Manager daemon x is now available 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: Queued rgw.foo for migration 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: Queued rgw.smpl for migration 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:35.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:35.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: Upgrade: Need to upgrade myself (mgr.y) 2024-03-26T04:40:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: Failing over to other MGR 2024-03-26T04:40:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:40:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.14448 172.21.15.17:0/4024413595' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: mgrmap e22: x(active, starting, since 0.999697s), standbys: y 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:40:35.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: Manager daemon x is now available 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: Queued rgw.foo for migration 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: Queued rgw.smpl for migration 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:40:35.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:35.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:36.115 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:35.940Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:36.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:35.787+0000 7f0860d12000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:40:36.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:35.879+0000 7f0860d12000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:40:36.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:35.964+0000 7f0860d12000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:40:36.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:36.114+0000 7f0860d12000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:40:36.396 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:36.199+0000 7f0860d12000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:40:36.396 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:36.310+0000 7f0860d12000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:36.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[29581]: Deploying cephadm binary to smithi129 2024-03-26T04:40:36.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[29581]: Deploying cephadm binary to smithi017 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[29581]: mgrmap e23: x(active, since 1.99868s), standbys: y 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[29581]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[24814]: Deploying cephadm binary to smithi129 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[24814]: Deploying cephadm binary to smithi017 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[24814]: mgrmap e23: x(active, since 1.99868s), standbys: y 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-mon[24814]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:36.396+0000 7f0860d12000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:40:36.655 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:36.566+0000 7f0860d12000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:36 smithi129 ceph-mon[25657]: Deploying cephadm binary to smithi129 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:36 smithi129 ceph-mon[25657]: Deploying cephadm binary to smithi017 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:36 smithi129 ceph-mon[25657]: mgrmap e23: x(active, since 1.99868s), standbys: y 2024-03-26T04:40:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:36 smithi129 ceph-mon[25657]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:36.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:36.654+0000 7f0860d12000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:40:37.846 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:37.711+0000 7f0860d12000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:40:37.846 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[24814]: [26/Mar/2024:04:40:36] ENGINE Bus STARTING 2024-03-26T04:40:37.846 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[24814]: [26/Mar/2024:04:40:36] ENGINE Serving on http://172.21.15.129:8765 2024-03-26T04:40:37.846 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[24814]: [26/Mar/2024:04:40:36] ENGINE Serving on https://172.21.15.129:7150 2024-03-26T04:40:37.846 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[24814]: [26/Mar/2024:04:40:36] ENGINE Bus STARTED 2024-03-26T04:40:37.846 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[24814]: mgrmap e24: x(active, since 3s), standbys: y 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[24814]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[29581]: [26/Mar/2024:04:40:36] ENGINE Bus STARTING 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[29581]: [26/Mar/2024:04:40:36] ENGINE Serving on http://172.21.15.129:8765 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[29581]: [26/Mar/2024:04:40:36] ENGINE Serving on https://172.21.15.129:7150 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[29581]: [26/Mar/2024:04:40:36] ENGINE Bus STARTED 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[29581]: mgrmap e24: x(active, since 3s), standbys: y 2024-03-26T04:40:37.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-mon[29581]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:37.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:37 smithi129 ceph-mon[25657]: [26/Mar/2024:04:40:36] ENGINE Bus STARTING 2024-03-26T04:40:37.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:37 smithi129 ceph-mon[25657]: [26/Mar/2024:04:40:36] ENGINE Serving on http://172.21.15.129:8765 2024-03-26T04:40:37.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:37 smithi129 ceph-mon[25657]: [26/Mar/2024:04:40:36] ENGINE Serving on https://172.21.15.129:7150 2024-03-26T04:40:37.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:37 smithi129 ceph-mon[25657]: [26/Mar/2024:04:40:36] ENGINE Bus STARTED 2024-03-26T04:40:37.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:37 smithi129 ceph-mon[25657]: mgrmap e24: x(active, since 3s), standbys: y 2024-03-26T04:40:37.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:37 smithi129 ceph-mon[25657]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:38.135 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:37.845+0000 7f0860d12000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:40:38.135 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:38.044+0000 7f0860d12000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:40:38.457 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:38.135+0000 7f0860d12000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:40:38.458 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:38.276+0000 7f0860d12000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:38.925 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:38.605+0000 7f0860d12000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:40:38.925 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:38.836+0000 7f0860d12000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:40:39.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:38.924+0000 7f0860d12000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:40:39.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:39.101+0000 7f0860d12000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:40:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:39 smithi129 ceph-mon[25657]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:39 smithi129 ceph-mon[25657]: mgrmap e25: x(active, since 5s), standbys: y 2024-03-26T04:40:39.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:39 smithi017 ceph-mon[24814]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:39.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:39 smithi017 ceph-mon[24814]: mgrmap e25: x(active, since 5s), standbys: y 2024-03-26T04:40:39.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:39 smithi017 ceph-mon[29581]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:39.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:39 smithi017 ceph-mon[29581]: mgrmap e25: x(active, since 5s), standbys: y 2024-03-26T04:40:40.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:39] "GET /metrics HTTP/1.1" 200 35349 "" "Prometheus/2.33.4" 2024-03-26T04:40:40.401 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:40.061+0000 7f0860d12000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:40:40.663 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:40.401+0000 7f0860d12000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:40:40.663 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:40.502+0000 7f0860d12000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:40:40.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:40.662+0000 7f0860d12000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:40:40.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:40:40] ENGINE Bus STARTING 2024-03-26T04:40:40.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: CherryPy Checker: 2024-03-26T04:40:40.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: The Application mounted at '' has an empty config. 2024-03-26T04:40:40.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: 2024-03-26T04:40:40.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:40:40] ENGINE Serving on http://:::9283 2024-03-26T04:40:40.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: [26/Mar/2024:04:40:40] ENGINE Bus STARTED 2024-03-26T04:40:41.205 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:40.946Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:41.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:41.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: Standby manager daemon y restarted 2024-03-26T04:40:41.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: Standby manager daemon y started 2024-03-26T04:40:41.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: Standby manager daemon y restarted 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: Standby manager daemon y started 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T04:40:41.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:41 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: Standby manager daemon y restarted 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: Standby manager daemon y started 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T04:40:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:41 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/1776212096' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:40:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:42 smithi129 ceph-mon[25657]: mgrmap e26: x(active, since 7s), standbys: y 2024-03-26T04:40:42.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:42 smithi017 ceph-mon[24814]: mgrmap e26: x(active, since 7s), standbys: y 2024-03-26T04:40:42.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:42 smithi017 ceph-mon[29581]: mgrmap e26: x(active, since 7s), standbys: y 2024-03-26T04:40:43.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:43 smithi129 ceph-mon[25657]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:40:43.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:43 smithi017 ceph-mon[29581]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:40:43.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:43 smithi017 ceph-mon[24814]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:40:43.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:40:43.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:43.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:43.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:44 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:44 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:44 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:44 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:44 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:44 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:45 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:40:45.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:40:45.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:45.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:45 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:46.808 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:46 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi017 to 2737M 2024-03-26T04:40:46.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:46 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi017 to 2737M 2024-03-26T04:40:46.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:46 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi017 to 2737M 2024-03-26T04:40:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:47 smithi129 ceph-mon[25657]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:40:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:47 smithi017 ceph-mon[24814]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:40:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:47 smithi017 ceph-mon[29581]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:40:48.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:48] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:40:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: Adjusting osd_memory_target on smithi129 to 2992M 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: Adjusting osd_memory_target on smithi129 to 2992M 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:40:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: Adjusting osd_memory_target on smithi129 to 2992M 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:49.709 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:49 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.205 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:49 smithi017 systemd[1]: Stopping Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:40:50.205 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:49 smithi017 bash[58403]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter.a" found: no such container 2024-03-26T04:40:50.205 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:49 smithi017 podman[58415]: 2024-03-26 04:40:49.9695719 +0000 UTC m=+0.117874751 container died af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:50.322 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:49] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-03-26T04:40:50.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:50.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:50.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:40:50.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: Reconfiguring daemon node-exporter.a on smithi017 2024-03-26T04:40:50.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:50 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:50.705 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 podman[58415]: 2024-03-26 04:40:50.409016201 +0000 UTC m=+0.557319036 container cleanup af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:50.705 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 bash[58415]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: Reconfiguring daemon node-exporter.a on smithi017 2024-03-26T04:40:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: Reconfiguring daemon node-exporter.a on smithi017 2024-03-26T04:40:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:50 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:50.999 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 podman[58429]: 2024-03-26 04:40:50.736552961 +0000 UTC m=+0.767452526 container remove af8933711de08a795b633f212e03f3fa23b693b577dfff704975021b285de2f7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:50.999 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 bash[58441]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter.a" found: no such container 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 systemd[1]: Stopped Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 2.675s CPU time. 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 systemd[1]: Starting Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:50 smithi017 podman[58511]: 2024-03-26 04:40:50.969564373 +0000 UTC m=+0.085371487 container create d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:51.000 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 podman[58511]: 2024-03-26 04:40:50.90232258 +0000 UTC m=+0.018129666 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T04:40:51.296 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 podman[58511]: 2024-03-26 04:40:51.205585349 +0000 UTC m=+0.321392427 container init d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:51.297 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 podman[58511]: 2024-03-26 04:40:51.208474221 +0000 UTC m=+0.324281290 container start d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:51.297 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.211Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T04:40:51.297 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.211Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T04:40:51.297 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T04:40:51.298 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T04:40:51.299 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.212Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T04:40:51.300 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a[58525]: ts=2024-03-26T04:40:51.213Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T04:40:51.301 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 bash[58511]: d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a 2024-03-26T04:40:51.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:51 smithi129 ceph-mon[25657]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-mon[24814]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:51.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:51 smithi017 ceph-mon[29581]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:51.705 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:40:51 smithi017 systemd[1]: Started Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:40:51.845 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:40:52.528 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:40:52.528 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 8s ago 4m 27.4M - 0.23.0 ba2b418f427c f2a5289b41b6 2024-03-26T04:40:52.528 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (4m) 8s ago 4m 53.6M - 8.3.5 dad864ee21e9 47e825a337c6 2024-03-26T04:40:52.528 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (3m) 8s ago 3m 53.8M - 3.5 e1d6a67b021e a7492721d53e 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283 running (38s) 8s ago 9m 491M - 19.0.0-1775-g5e70124b 5f21c9264136 7f62fb476af1 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (10m) 8s ago 10m 381M - 17.2.0 e1d6a67b021e 9b468d7472cd 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (10m) 8s ago 10m 50.7M 2048M 17.2.0 e1d6a67b021e e512d90ccc03 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (9m) 8s ago 9m 44.0M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (9m) 8s ago 9m 42.0M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 starting - - - - 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (5m) 8s ago 5m 16.2M - 1.3.1 1dbe0e931976 d147c628643f 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (8m) 8s ago 8m 52.0M 2737M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (8m) 8s ago 8m 55.2M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (8m) 8s ago 8m 50.7M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (7m) 8s ago 7m 53.3M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (7m) 8s ago 7m 54.3M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (6m) 8s ago 6m 50.3M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (6m) 8s ago 6m 54.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (6m) 8s ago 6m 52.2M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:40:52.529 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (4m) 8s ago 5m 67.1M - 2.33.4 514e6a882f6e faf1d87a92d7 2024-03-26T04:40:52.530 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (4m) 8s ago 4m 71.0M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:40:52.530 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (3m) 8s ago 3m 74.8M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:40:52.530 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (3m) 8s ago 3m 71.5M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:40:52.530 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (3m) 8s ago 3m 72.1M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:40:52.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:40:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:40:52.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.129:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:40:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:52 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:52 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:52 smithi129 ceph-mon[25657]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:40:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:52 smithi129 ceph-mon[25657]: Reconfiguring daemon alertmanager.a on smithi017 2024-03-26T04:40:52.653 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:52.653 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:52.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[29581]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:40:52.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[29581]: Reconfiguring daemon alertmanager.a on smithi017 2024-03-26T04:40:52.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:52.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:40:52.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[24814]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:40:52.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:52 smithi017 ceph-mon[24814]: Reconfiguring daemon alertmanager.a on smithi017 2024-03-26T04:40:52.943 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:40:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:40:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T04:40:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:40:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:40:52.945 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:40:53.331 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:40:53.331 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:40:53.331 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:40:53.331 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:40:53.331 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-03-26T04:40:53.332 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/23 daemons upgraded", 2024-03-26T04:40:53.332 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-03-26T04:40:53.332 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:40:53.332 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:40:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:53 smithi129 ceph-mon[25657]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:53 smithi129 ceph-mon[25657]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:53 smithi129 ceph-mon[25657]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:53 smithi129 ceph-mon[25657]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:53 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2469284308' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:40:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[24814]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[24814]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[24814]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[24814]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2469284308' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:40:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[29581]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[29581]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[29581]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[29581]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:40:53.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2469284308' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:40:53.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=error ts=2024-03-26T04:40:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:53.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.129:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.129 because it doesn't contain any IP SANs" 2024-03-26T04:40:53.706 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=warn ts=2024-03-26T04:40:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-03-26T04:40:53.786 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:40:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:54 smithi129 ceph-mon[25657]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:54 smithi129 ceph-mon[25657]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:54 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1092895479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:40:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:54 smithi017 ceph-mon[24814]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:54 smithi017 ceph-mon[24814]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:54 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1092895479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:40:54.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:54 smithi017 ceph-mon[29581]: from='client.24772 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:54.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:54 smithi017 ceph-mon[29581]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:40:54.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:54 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1092895479' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:40:55.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:55 smithi129 ceph-mon[25657]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:55.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:55 smithi017 ceph-mon[24814]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:55.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:55 smithi017 ceph-mon[29581]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:57.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:57 smithi017 ceph-mon[24814]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:57.677 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:57 smithi017 ceph-mon[29581]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:57.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:57 smithi129 ceph-mon[25657]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:58.663 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:40:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:40:58.955 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:58 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:40:58.956 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:58 smithi017 bash[58926]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T04:40:59.275 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[49488]: level=info ts=2024-03-26T04:40:58.959Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:40:59.275 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:58 smithi017 podman[58938]: 2024-03-26 04:40:58.980348909 +0000 UTC m=+0.148546434 container died f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:59.544 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 podman[58938]: 2024-03-26 04:40:59.368262713 +0000 UTC m=+0.536460288 container cleanup f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:59.544 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 bash[58938]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:40:59.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:40:59 smithi017 ceph-mon[24814]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:59.863 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:40:59 smithi017 ceph-mon[29581]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:59.864 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 podman[58952]: 2024-03-26 04:40:59.544334794 +0000 UTC m=+0.563563440 container remove f2a5289b41b631082bcce3d9261fa9a0e5dd518f048bf87cecef3f1439572ef3 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:40:59.864 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 podman[58952]: 2024-03-26 04:40:59.611643868 +0000 UTC m=+0.630872471 volume remove d9cb0cc23e77d2bd67ce0a58e042adc1b583f6effd13e609cbac724ac9ee73e5 2024-03-26T04:40:59.864 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 bash[58966]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T04:40:59.864 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:40:59.864 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:40:59.864 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 systemd[1]: Starting Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:40:59.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:40:59 smithi129 ceph-mon[25657]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:00.189 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 podman[59031]: 2024-03-26 04:40:59.766554551 +0000 UTC m=+0.017821001 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-26T04:41:00.189 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 podman[59031]: 2024-03-26 04:40:59.887462958 +0000 UTC m=+0.138729393 volume create 34412dce14c8f7bcd47243475f46f4c6e73cd3020f8abed952fe413111e0bdfe 2024-03-26T04:41:00.189 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:40:59 smithi017 podman[59031]: 2024-03-26 04:40:59.996382312 +0000 UTC m=+0.247648789 container create 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:00.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:40:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:40:59] "GET /metrics HTTP/1.1" 200 38154 "" "Prometheus/2.33.4" 2024-03-26T04:41:00.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 podman[59031]: 2024-03-26 04:41:00.282592113 +0000 UTC m=+0.533858550 container init 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:00.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 podman[59031]: 2024-03-26 04:41:00.285202329 +0000 UTC m=+0.536468765 container start 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:00.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.295Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.295Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.297Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.298Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.324Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.324Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.326Z caller=main.go:518 msg=Listening address=:9093 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:00.326Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 bash[59031]: 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f 2024-03-26T04:41:00.456 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:00 smithi017 systemd[1]: Started Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:01.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: Reconfiguring iscsi.foo.smithi017.atsgyu (dependencies changed)... 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: Reconfiguring daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[29581]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:01.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:01.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: Reconfiguring iscsi.foo.smithi017.atsgyu (dependencies changed)... 2024-03-26T04:41:01.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:01.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:01.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:01.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: Reconfiguring daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:41:01.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:01 smithi017 ceph-mon[24814]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: Reconfiguring iscsi.foo.smithi017.atsgyu (dependencies changed)... 2024-03-26T04:41:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:01.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: Reconfiguring daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:41:01.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:01 smithi129 ceph-mon[25657]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:02.455 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:02 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:02.299Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000972793s 2024-03-26T04:41:02.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:02.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.129:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:41:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-mon[25657]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:03.898 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:03 smithi017 ceph-mon[24814]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:03.899 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:03 smithi017 ceph-mon[29581]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:03.975+0000 7fae7b50a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Creating ceph-iscsi config... 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Traceback (most recent call last): 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:04.396 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return _run_code(code, main_globals, None, 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: exec(code, run_globals) 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:04.397 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Traceback (most recent call last): 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return self.event_loop.get_result(coro, timeout) 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-26T04:41:04.398 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return future.result(timeout) 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return self.__get_result() 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: raise self._exception 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: config_blobs=daemon_spec.final_config, 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Creating ceph-iscsi config... 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:04.399 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Traceback (most recent call last): 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return _run_code(code, main_globals, None, 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: exec(code, run_globals) 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:04.400 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:03 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:05 smithi129 ceph-mon[25657]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout: Creating ceph-iscsi config... 2024-03-26T04:41:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: exec(code, run_globals) 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: return future.result(timeout) 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.__get_result() 2024-03-26T04:41:05.897 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: raise self._exception 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: Creating ceph-iscsi config... 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:05.898 INFO:journalctl@ceph.mon.b.smithi129.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: exec(code, run_globals) 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:05 smithi129 ceph-mon[25657]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:05 smithi129 ceph-mon[25657]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:41:05.899 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:05 smithi129 ceph-mon[25657]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:05.900 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:05 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:05.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[24814]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-26T04:41:05.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:05.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:05.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[24814]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[24814]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[24814]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[29581]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:05.960 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:05.961 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T04:41:05.962 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:05.963 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[29581]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[29581]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[29581]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:05.964 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:05 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:07.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:07 smithi129 ceph-mon[25657]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:41:07.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:07 smithi017 ceph-mon[24814]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:41:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:07 smithi017 ceph-mon[29581]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:41:08.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[25042]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:41:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:41:09.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:09 smithi129 ceph-mon[25657]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:09 smithi017 ceph-mon[24814]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:09 smithi017 ceph-mon[29581]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:10.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:09 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:41:09] "GET /metrics HTTP/1.1" 200 38152 "" "Prometheus/2.33.4" 2024-03-26T04:41:10.705 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:41:10 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:41:10.302Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004552632s 2024-03-26T04:41:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:11 smithi129 ceph-mon[25657]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:11.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:11 smithi017 ceph-mon[24814]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:11.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:11 smithi017 ceph-mon[29581]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:12.395 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:12 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.129:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:41:13.890 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:13 smithi129 ceph-mon[25657]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:13.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:13 smithi017 ceph-mon[29581]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:13.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:13 smithi017 ceph-mon[24814]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:41:14.519 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 systemd[1]: Stopping Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:14.519 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 bash[50304]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:41:14.804 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.519Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.519Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.519Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.519Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.519Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.519Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.520Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.520Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.520Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.520Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.520Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[42554]: ts=2024-03-26T04:41:14.520Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T04:41:14.805 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 podman[50317]: 2024-03-26 04:41:14.529825883 +0000 UTC m=+0.173642401 container died faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:15.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 podman[50317]: 2024-03-26 04:41:14.894068566 +0000 UTC m=+0.537885093 container cleanup faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:15.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 bash[50317]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:41:15.146 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:14 smithi129 bash[50346]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:41:15.483 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 podman[50331]: 2024-03-26 04:41:15.162127645 +0000 UTC m=+0.628928814 container remove faf1d87a92d76cc22b9e0aa861c90bf325f4876d96d28b4f9ca7ca49bbae8c0f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:15.483 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T04:41:15.483 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 systemd[1]: Stopped Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:15.483 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.506s CPU time. 2024-03-26T04:41:15.483 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 systemd[1]: Starting Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:15.484 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 podman[50413]: 2024-03-26 04:41:15.305085652 +0000 UTC m=+0.019012597 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T04:41:15.484 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 podman[50413]: 2024-03-26 04:41:15.404650244 +0000 UTC m=+0.118577232 container create 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:15.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-mon[25657]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:41:15.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3054474638' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 podman[50413]: 2024-03-26 04:41:15.598852983 +0000 UTC m=+0.312779927 container init 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 podman[50413]: 2024-03-26 04:41:15.601650594 +0000 UTC m=+0.315577547 container start 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.625Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.625Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.625Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.625Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi129 (none))" 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.625Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:41:15.824 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.625Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.627Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.627Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.628Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.630Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.630Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.371µs 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.630Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 bash[50413]: 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 systemd[1]: Started Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:15.825 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:15.824Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-26T04:41:15.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:15 smithi017 ceph-mon[29581]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:41:15.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:15 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3054474638' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:41:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:15 smithi017 ceph-mon[24814]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:41:15.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:15 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3054474638' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.056Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.056Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.056Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.085µs wal_replay_duration=426.322117ms total_replay_duration=426.365872ms 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.059Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.059Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.059Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:41:16.076 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.076Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.748146ms db_storage=719ns remote_storage=1.32µs web_handler=302ns query_engine=681ns scrape=408.023µs scrape_sd=78.765µs notify=11.789µs notify_sd=25.331µs rules=15.918842ms 2024-03-26T04:41:16.077 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:16.076Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T04:41:16.378 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:16 smithi129 systemd[1]: Stopping Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:16.378 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:16 smithi129 bash[50536]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter.b" found: no such container 2024-03-26T04:41:16.645 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:16 smithi129 podman[50549]: 2024-03-26 04:41:16.378692312 +0000 UTC m=+0.115586223 container died d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:41:16.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:16.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:16.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[29581]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-26T04:41:16.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[29581]: Reconfiguring daemon node-exporter.b on smithi129 2024-03-26T04:41:16.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2532614793' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3507232506"}]: dispatch 2024-03-26T04:41:16.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[29581]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T04:41:16.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:16.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:16.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[24814]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-26T04:41:16.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[24814]: Reconfiguring daemon node-exporter.b on smithi129 2024-03-26T04:41:16.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2532614793' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3507232506"}]: dispatch 2024-03-26T04:41:16.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:16 smithi017 ceph-mon[24814]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T04:41:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-mon[25657]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-26T04:41:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-mon[25657]: Reconfiguring daemon node-exporter.b on smithi129 2024-03-26T04:41:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2532614793' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3507232506"}]: dispatch 2024-03-26T04:41:17.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:16 smithi129 ceph-mon[25657]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T04:41:18.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:17 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2532614793' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3507232506"}]': finished 2024-03-26T04:41:18.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:17 smithi129 ceph-mon[25657]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:41:18.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:17 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1796910915' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/912684274"}]: dispatch 2024-03-26T04:41:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:17 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2532614793' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3507232506"}]': finished 2024-03-26T04:41:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:17 smithi017 ceph-mon[24814]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:41:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:17 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1796910915' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/912684274"}]: dispatch 2024-03-26T04:41:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:17 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2532614793' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3507232506"}]': finished 2024-03-26T04:41:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:17 smithi017 ceph-mon[29581]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:41:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:17 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1796910915' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/912684274"}]: dispatch 2024-03-26T04:41:18.504 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 podman[50549]: 2024-03-26 04:41:18.245386868 +0000 UTC m=+1.982280779 container cleanup d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:41:18.504 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 bash[50549]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T04:41:18.504 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 bash[50576]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter.b" found: no such container 2024-03-26T04:41:18.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1796910915' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/912684274"}]': finished 2024-03-26T04:41:18.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-mon[25657]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:41:18.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3464574335' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]: dispatch 2024-03-26T04:41:18.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]: dispatch 2024-03-26T04:41:18.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-mon[25657]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 podman[50564]: 2024-03-26 04:41:18.504915363 +0000 UTC m=+2.130672513 container remove d147c628643f0bfba68c000fe7182894829792662aac9c7060d30e5043afc1b7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 systemd[1]: Stopped Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 2.814s CPU time. 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 systemd[1]: Starting Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 podman[50649]: 2024-03-26 04:41:18.643191521 +0000 UTC m=+0.018329106 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T04:41:18.773 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 podman[50649]: 2024-03-26 04:41:18.77284705 +0000 UTC m=+0.147984657 container create 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:41:19.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 podman[50649]: 2024-03-26 04:41:18.950255658 +0000 UTC m=+0.325393223 container init 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:41:19.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 podman[50649]: 2024-03-26 04:41:18.952745971 +0000 UTC m=+0.327883546 container start 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:41:19.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.955Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T04:41:19.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.955Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T04:41:19.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T04:41:19.042 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T04:41:19.043 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.956Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T04:41:19.044 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b[50663]: ts=2024-03-26T04:41:18.957Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T04:41:19.045 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:19 smithi129 bash[50649]: 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 2024-03-26T04:41:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1796910915' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/912684274"}]': finished 2024-03-26T04:41:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[24814]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:41:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3464574335' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]: dispatch 2024-03-26T04:41:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]: dispatch 2024-03-26T04:41:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[24814]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-26T04:41:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1796910915' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/912684274"}]': finished 2024-03-26T04:41:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[29581]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:41:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3464574335' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]: dispatch 2024-03-26T04:41:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]: dispatch 2024-03-26T04:41:19.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:18 smithi017 ceph-mon[29581]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-26T04:41:19.395 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:41:19 smithi129 systemd[1]: Started Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]': finished 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/368833640' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]: dispatch 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]: dispatch 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: Reconfiguring grafana.a (dependencies changed)... 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:41:20.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: Reconfiguring daemon grafana.a on smithi129 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]': finished 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/368833640' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]: dispatch 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]: dispatch 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: Reconfiguring grafana.a (dependencies changed)... 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: Reconfiguring daemon grafana.a on smithi129 2024-03-26T04:41:20.096 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:19 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3616639726"}]': finished 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/368833640' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]: dispatch 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]: dispatch 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: Reconfiguring grafana.a (dependencies changed)... 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:20.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:20.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:41:20.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: Reconfiguring daemon grafana.a on smithi129 2024-03-26T04:41:20.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:19 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:21 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]': finished 2024-03-26T04:41:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:21 smithi129 ceph-mon[25657]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:41:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:21 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2056448767' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1340511436"}]: dispatch 2024-03-26T04:41:21.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:21 smithi129 ceph-mon[25657]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]': finished 2024-03-26T04:41:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[24814]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:41:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2056448767' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1340511436"}]: dispatch 2024-03-26T04:41:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[24814]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2685930724"}]': finished 2024-03-26T04:41:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[29581]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:41:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2056448767' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1340511436"}]: dispatch 2024-03-26T04:41:21.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:21 smithi017 ceph-mon[29581]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:22 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2056448767' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1340511436"}]': finished 2024-03-26T04:41:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:22 smithi129 ceph-mon[25657]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:41:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:22 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1062778490' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]: dispatch 2024-03-26T04:41:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:22 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]: dispatch 2024-03-26T04:41:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2056448767' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1340511436"}]': finished 2024-03-26T04:41:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[24814]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:41:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1062778490' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]: dispatch 2024-03-26T04:41:22.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]: dispatch 2024-03-26T04:41:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2056448767' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1340511436"}]': finished 2024-03-26T04:41:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[29581]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:41:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1062778490' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]: dispatch 2024-03-26T04:41:22.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:22 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]: dispatch 2024-03-26T04:41:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:23 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]': finished 2024-03-26T04:41:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:23 smithi129 ceph-mon[25657]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:41:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:23 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2150992442' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]: dispatch 2024-03-26T04:41:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:23 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]: dispatch 2024-03-26T04:41:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:23 smithi129 ceph-mon[25657]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:23.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]': finished 2024-03-26T04:41:23.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[24814]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:41:23.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2150992442' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]: dispatch 2024-03-26T04:41:23.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]: dispatch 2024-03-26T04:41:23.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[24814]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:23.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1340511436"}]': finished 2024-03-26T04:41:23.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[29581]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:41:23.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2150992442' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]: dispatch 2024-03-26T04:41:23.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]: dispatch 2024-03-26T04:41:23.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:23 smithi017 ceph-mon[29581]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:24.156 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:41:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:24 smithi129 ceph-mon[25657]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]': finished 2024-03-26T04:41:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:24 smithi129 ceph-mon[25657]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:41:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:24 smithi017 ceph-mon[29581]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]': finished 2024-03-26T04:41:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:24 smithi017 ceph-mon[29581]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:41:24.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:24 smithi017 ceph-mon[24814]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3847572043"}]': finished 2024-03-26T04:41:24.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:24 smithi017 ceph-mon[24814]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:41:24.820 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:41:24.820 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 starting - - - - 2024-03-26T04:41:24.820 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (5m) 40s ago 5m 53.6M - 8.3.5 dad864ee21e9 47e825a337c6 2024-03-26T04:41:24.820 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (4m) 41s ago 4m 53.8M - 3.5 e1d6a67b021e a7492721d53e 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283 running (70s) 40s ago 9m 491M - 19.0.0-1775-g5e70124b 5f21c9264136 7f62fb476af1 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (11m) 41s ago 11m 381M - 17.2.0 e1d6a67b021e 9b468d7472cd 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (11m) 41s ago 11m 50.7M 2048M 17.2.0 e1d6a67b021e e512d90ccc03 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (9m) 40s ago 9m 44.0M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (10m) 41s ago 10m 42.0M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 starting - - - - 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 starting - - - - 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (9m) 41s ago 9m 52.0M 2737M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (9m) 41s ago 9m 55.2M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (8m) 41s ago 8m 50.7M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (8m) 41s ago 8m 53.3M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (7m) 40s ago 7m 54.3M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (7m) 40s ago 7m 50.3M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (7m) 40s ago 7m 54.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (6m) 40s ago 6m 52.2M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 starting - - - - 2024-03-26T04:41:24.821 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (4m) 41s ago 4m 71.0M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:41:24.822 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (4m) 40s ago 4m 74.8M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:41:24.822 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (4m) 41s ago 4m 71.5M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:41:24.822 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (4m) 40s ago 4m 72.1M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:41:25.229 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:41:25.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:25.231 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:41:25.231 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-03-26T04:41:25.231 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:41:25.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:41:25.231 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:25 smithi129 ceph-mon[25657]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:25 smithi129 ceph-mon[25657]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:25.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:25 smithi129 ceph-mon[25657]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:41:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:25 smithi017 ceph-mon[24814]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:25 smithi017 ceph-mon[24814]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:25 smithi017 ceph-mon[24814]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:41:25.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:25 smithi017 ceph-mon[29581]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:25.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:25 smithi017 ceph-mon[29581]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:25.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:25 smithi017 ceph-mon[29581]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/23 daemons upgraded", 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:41:25.573 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:41:25.972 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:41:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:26 smithi129 ceph-mon[25657]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:26 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3117260422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:41:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:26 smithi129 ceph-mon[25657]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:26 smithi129 ceph-mon[25657]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:26.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:26 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2882688118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:41:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[24814]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3117260422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:41:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[24814]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[24814]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/2882688118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:41:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[29581]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3117260422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:41:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[29581]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[29581]: from='client.25204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:26.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:26 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2882688118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:41:27.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:27 smithi129 ceph-mon[25657]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:27 smithi017 ceph-mon[24814]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:27.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:27 smithi017 ceph-mon[29581]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:29.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:41:29] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.33.4" 2024-03-26T04:41:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:29 smithi129 ceph-mon[25657]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-26T04:41:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:29 smithi017 ceph-mon[24814]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-26T04:41:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:29 smithi017 ceph-mon[29581]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-03-26T04:41:31.335 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:31 smithi129 ceph-mon[25657]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:31.335 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 systemd[1]: Stopping Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:31.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:31 smithi017 ceph-mon[24814]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:31.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:31 smithi017 ceph-mon[29581]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:31.638 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 bash[50859]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana.a" found: no such container 2024-03-26T04:41:31.638 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[41753]: t=2024-03-26T04:41:31+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-26T04:41:31.638 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 podman[50871]: 2024-03-26 04:41:31.454482619 +0000 UTC m=+0.109894081 container died 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base rhel8, name=ubi8, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.buildah.version=1.24.2, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.openshift.expose-services=) 2024-03-26T04:41:31.895 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 podman[50871]: 2024-03-26 04:41:31.776933517 +0000 UTC m=+0.432344978 container cleanup 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, architecture=x86_64, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182) 2024-03-26T04:41:31.895 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 bash[50871]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a 2024-03-26T04:41:31.895 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 bash[50899]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana.a" found: no such container 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 podman[50885]: 2024-03-26 04:41:31.995244836 +0000 UTC m=+0.540974174 container remove 47e825a337c695df0f20e7ec11dd0089d889feb5465292279a093a6f69701b65 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.openshift.expose-services=, name=ubi8, com.redhat.component=ubi8-container, release=236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 systemd[1]: Stopped Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:31 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@grafana.a.service: Consumed 2.572s CPU time. 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 systemd[1]: Starting Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 podman[50979]: 2024-03-26 04:41:32.135729438 +0000 UTC m=+0.018787347 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T04:41:32.246 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 podman[50979]: 2024-03-26 04:41:32.246448843 +0000 UTC m=+0.129506747 container create 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, release=236.1648460182, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, architecture=x86_64, maintainer=Paul Cuzner , version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8) 2024-03-26T04:41:32.499 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 podman[50979]: 2024-03-26 04:41:32.44891608 +0000 UTC m=+0.331973984 container init 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, maintainer=Paul Cuzner , io.openshift.expose-services=, version=8.5, name=ubi8, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T04:41:32.499 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 podman[50979]: 2024-03-26 04:41:32.451743275 +0000 UTC m=+0.334801188 container start 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, architecture=x86_64, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, name=ubi8, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182) 2024-03-26T04:41:32.499 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-26T04:41:32.500 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-26T04:41:32.501 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-26T04:41:32.501 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="App mode production" logger=settings 2024-03-26T04:41:32.501 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-26T04:41:32.501 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T04:41:32.501 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-26T04:41:32.844 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STOPPING 2024-03-26T04:41:32.844 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:41:32.844 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STOPPED 2024-03-26T04:41:32.844 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STARTING 2024-03-26T04:41:32.844 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Serving on http://:::9283 2024-03-26T04:41:32.844 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STARTED 2024-03-26T04:41:32.845 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STOPPING 2024-03-26T04:41:32.845 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:41:32.845 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STOPPED 2024-03-26T04:41:32.845 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STARTING 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=554.269µs 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 bash[50979]: 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 systemd[1]: Started Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:32.845 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-26T04:41:32.846 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:41:32+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T04:41:33.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Serving on http://:::9283 2024-03-26T04:41:33.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STARTED 2024-03-26T04:41:33.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STOPPING 2024-03-26T04:41:33.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:41:33.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STOPPED 2024-03-26T04:41:33.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STARTING 2024-03-26T04:41:33.146 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Serving on http://:::9283 2024-03-26T04:41:33.146 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:32] ENGINE Bus STARTED 2024-03-26T04:41:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:41:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:33.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T04:41:33.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:33 smithi129 ceph-mon[25657]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T04:41:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:41:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:41:33.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T04:41:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[24814]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:41:33.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T04:41:33.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:33 smithi017 ceph-mon[29581]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: Upgrade: Updating mgr.y 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: Deploying daemon mgr.y on smithi017 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:35.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: Upgrade: Updating mgr.y 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: Deploying daemon mgr.y on smithi017 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-26T04:41:35.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:35 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: Upgrade: Updating mgr.y 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:35.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: Deploying daemon mgr.y on smithi017 2024-03-26T04:41:35.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-26T04:41:35.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:35 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:36.314 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:36 smithi017 ceph-mon[24814]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:36.314 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:36 smithi017 systemd[1]: Stopping Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:36.314 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:36 smithi017 ceph-mon[29581]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:36.607 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:36 smithi017 podman[60136]: 2024-03-26 04:41:36.313310172 +0000 UTC m=+0.136825804 container died 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, RELEASE=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_CLEAN=True, name=centos-stream, release=754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, distribution-scope=public, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-03-26T04:41:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:36 smithi129 ceph-mon[25657]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:36.933 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:36 smithi017 podman[60136]: 2024-03-26 04:41:36.682343343 +0000 UTC m=+0.505858966 container cleanup 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_CLEAN=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, distribution-scope=public, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64) 2024-03-26T04:41:36.933 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:36 smithi017 bash[60136]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y 2024-03-26T04:41:37.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:36 smithi017 podman[60150]: 2024-03-26 04:41:36.933861614 +0000 UTC m=+0.621749886 container remove 9b468d7472cd58506af5a28a220ba50ab42c612c4be4f711ab95709ab6e672ec (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True, vendor=Red Hat, Inc., version=8, distribution-scope=public, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , architecture=x86_64, RELEASE=HEAD, vcs-type=git, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8) 2024-03-26T04:41:37.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:36 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:41:37.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-26T04:41:37.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 systemd[1]: Stopped Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:37.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service: Consumed 56.549s CPU time. 2024-03-26T04:41:37.547 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:37 smithi017 ceph-mon[24814]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:37.548 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:37 smithi017 ceph-mon[29581]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:37.548 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 systemd[1]: Starting Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:41:37.548 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 podman[60278]: 2024-03-26 04:41:37.442946421 +0000 UTC m=+0.104917462 container create 4836c6142fed2f2f8ebfce0da45f9c0834c84beb8d012f821f4a49ae628ac017 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:41:37.548 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 podman[60278]: 2024-03-26 04:41:37.355380189 +0000 UTC m=+0.017351230 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:41:37.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:37 smithi129 ceph-mon[25657]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:37.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 podman[60278]: 2024-03-26 04:41:37.645175086 +0000 UTC m=+0.307146124 container init 4836c6142fed2f2f8ebfce0da45f9c0834c84beb8d012f821f4a49ae628ac017 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T04:41:37.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 podman[60278]: 2024-03-26 04:41:37.647705089 +0000 UTC m=+0.309676129 container start 4836c6142fed2f2f8ebfce0da45f9c0834c84beb8d012f821f4a49ae628ac017 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T04:41:37.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 ceph-mgr[60299]: -- 172.21.15.17:0/1984332485 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a81efca1a0 con 0x55a81efa0c00 2024-03-26T04:41:37.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 bash[60278]: 4836c6142fed2f2f8ebfce0da45f9c0834c84beb8d012f821f4a49ae628ac017 2024-03-26T04:41:37.956 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:37 smithi017 systemd[1]: Started Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:41:38.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:38.219+0000 7efcfa3eb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:41:38.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:38.323+0000 7efcfa3eb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:41:39.122 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.123 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.123 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.123 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.123 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.123 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.123 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-mon[25657]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:39.123 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:41:39.130 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.033+0000 7efcfa3eb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[24814]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.132 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.132 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.132 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.132 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:39.132 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:38 smithi017 ceph-mon[29581]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:39.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.130+0000 7efcfa3eb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:41:39.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.280+0000 7efcfa3eb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:41:39.381 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.380+0000 7efcfa3eb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:41:39.694 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.468+0000 7efcfa3eb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:41:39.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:41:39] "GET /metrics HTTP/1.1" 200 38149 "" "Prometheus/2.33.4" 2024-03-26T04:41:39.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.763+0000 7efcfa3eb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:41:39.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.884+0000 7efcfa3eb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:40.283 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:39 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:39.984+0000 7efcfa3eb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:41:40.284 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:40.086+0000 7efcfa3eb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:41:40.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:40 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:40.283+0000 7efcfa3eb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:41:41.293 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:41 smithi017 ceph-mon[24814]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:41.293 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:41 smithi017 ceph-mon[29581]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:41.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:41 smithi129 ceph-mon[25657]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:41.616 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:41 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:41.353+0000 7efcfa3eb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:41:41.616 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:41 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:41.492+0000 7efcfa3eb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:41:41.933 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:41 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:41.838+0000 7efcfa3eb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:41:42.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:41 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:41.933+0000 7efcfa3eb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:41:42.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:42 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:42.162+0000 7efcfa3eb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:42.587 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:42 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:42.358+0000 7efcfa3eb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:41:42.931 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:42 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:42.661+0000 7efcfa3eb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:42.932 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:42 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:42.931+0000 7efcfa3eb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:43.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:43.189+0000 7efcfa3eb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:41:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:43 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:43 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:43 smithi129 ceph-mon[25657]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:43 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:43 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[29581]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[24814]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:43.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:43 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:44.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:44.068+0000 7efcfa3eb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:41:44.908 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:44.690+0000 7efcfa3eb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:41:44.908 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:44.798+0000 7efcfa3eb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:41:45.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:45.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:45.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-mon[29581]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:44 smithi017 ceph-mon[24814]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:45.035+0000 7efcfa3eb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:45] ENGINE Bus STARTING 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: CherryPy Checker: 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: The Application mounted at '' has an empty config. 2024-03-26T04:41:45.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:41:45.207 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:45] ENGINE Serving on http://:::9283 2024-03-26T04:41:45.207 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:45] ENGINE Bus STARTED 2024-03-26T04:41:45.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:44 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:45.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:44 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:45.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:44 smithi129 ceph-mon[25657]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:46.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: Standby manager daemon y restarted 2024-03-26T04:41:46.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: Standby manager daemon y started 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[29581]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: Standby manager daemon y restarted 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: Standby manager daemon y started 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T04:41:46.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:46.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:45 smithi017 ceph-mon[24814]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: Standby manager daemon y restarted 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: Standby manager daemon y started 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:46.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:45 smithi129 ceph-mon[25657]: from='client.24968 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:47.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:47 smithi017 ceph-mon[24814]: mgrmap e27: x(active, since 72s), standbys: y 2024-03-26T04:41:47.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:47 smithi017 ceph-mon[24814]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:47 smithi017 ceph-mon[29581]: mgrmap e27: x(active, since 72s), standbys: y 2024-03-26T04:41:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:47 smithi017 ceph-mon[29581]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:47.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:47 smithi129 ceph-mon[25657]: mgrmap e27: x(active, since 72s), standbys: y 2024-03-26T04:41:47.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:47 smithi129 ceph-mon[25657]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: Reconfiguring iscsi.foo.smithi017.atsgyu (dependencies changed)... 2024-03-26T04:41:48.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:48.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:48.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:48.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-mon[25657]: Reconfiguring daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:41:48.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: Reconfiguring iscsi.foo.smithi017.atsgyu (dependencies changed)... 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[24814]: Reconfiguring daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:48.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' 2024-03-26T04:41:48.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: Reconfiguring iscsi.foo.smithi017.atsgyu (dependencies changed)... 2024-03-26T04:41:48.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:48.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:41:48.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:48.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:48 smithi017 ceph-mon[29581]: Reconfiguring daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:41:49.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:41:49.628 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:49 smithi017 ceph-mon[29581]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:49.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:49 smithi017 ceph-mon[24814]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:49.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:49 smithi129 ceph-mon[25657]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:49.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:41:49] "GET /metrics HTTP/1.1" 200 38149 "" "Prometheus/2.33.4" 2024-03-26T04:41:50.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:50 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:50 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:50.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:50 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:51.382 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:51.094+0000 7fae7b50a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:51.382 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Creating ceph-iscsi config... 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Traceback (most recent call last): 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return _run_code(code, main_globals, None, 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: exec(code, run_globals) 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:51.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Traceback (most recent call last): 2024-03-26T04:41:51.384 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return self.event_loop.get_result(coro, timeout) 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return future.result(timeout) 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return self.__get_result() 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: raise self._exception 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: config_blobs=daemon_spec.final_config, 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:41:51.385 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Creating ceph-iscsi config... 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Non-zero exit code 1 from systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: Traceback (most recent call last): 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: return _run_code(code, main_globals, None, 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: exec(code, run_globals) 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:41:51.386 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:41:51.387 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1065, in deploy_daemon 2024-03-26T04:41:51.387 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:41:51.387 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: RuntimeError: Failed command: systemctl restart ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:41:51.387 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:41:51.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[24814]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:51.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:51.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[24814]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T04:41:51.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T04:41:51.617 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[24814]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:51] ENGINE Bus STOPPING 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:51] ENGINE Bus STOPPED 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:51] ENGINE Bus STARTING 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[29581]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[29581]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T04:41:51.618 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-mon[29581]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-mon[25657]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-mon[25657]: from='mgr.24860 172.21.15.129:0/2938123871' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-mon[25657]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:51.387+0000 7faee8e96700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-26T04:41:51.633 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ignoring --setuser ceph since I am not root 2024-03-26T04:41:51.634 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ignoring --setgroup ceph since I am not root 2024-03-26T04:41:51.634 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-mgr[46787]: -- 172.21.15.129:0/1470355021 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592f35761a0 con 0x5592f354d000 2024-03-26T04:41:51.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:51] ENGINE Serving on http://:::9283 2024-03-26T04:41:51.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:41:51 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:41:51] ENGINE Bus STARTED 2024-03-26T04:41:52.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:51.960+0000 7fb04be08200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:41:52.383 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:52.060+0000 7fb04be08200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.24860 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: mgrmap e28: y(active, starting, since 0.278181s) 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:41:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: Manager daemon y is now available 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:52.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:52.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:52.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:52.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:52.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.24860 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-26T04:41:52.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: mgrmap e28: y(active, starting, since 0.278181s) 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:41:52.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: Manager daemon y is now available 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.24860 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: mgrmap e28: y(active, starting, since 0.278181s) 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:41:52.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: Manager daemon y is now available 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:52.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:41:52.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:52.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:52 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:41:52.993 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:52.754+0000 7fb04be08200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:41:52.994 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:52.848+0000 7fb04be08200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:41:52.994 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:52.993+0000 7fb04be08200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:41:53.392 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.092+0000 7fb04be08200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:41:53.392 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.176+0000 7fb04be08200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:41:53.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[29581]: [26/Mar/2024:04:41:51] ENGINE Bus STARTING 2024-03-26T04:41:53.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[29581]: [26/Mar/2024:04:41:51] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:41:53.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[29581]: [26/Mar/2024:04:41:52] ENGINE Serving on http://172.21.15.17:8765 2024-03-26T04:41:53.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[29581]: [26/Mar/2024:04:41:52] ENGINE Bus STARTED 2024-03-26T04:41:53.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[29581]: mgrmap e29: y(active, since 1.28366s) 2024-03-26T04:41:53.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[29581]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:53.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[24814]: [26/Mar/2024:04:41:51] ENGINE Bus STARTING 2024-03-26T04:41:53.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[24814]: [26/Mar/2024:04:41:51] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:41:53.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[24814]: [26/Mar/2024:04:41:52] ENGINE Serving on http://172.21.15.17:8765 2024-03-26T04:41:53.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[24814]: [26/Mar/2024:04:41:52] ENGINE Bus STARTED 2024-03-26T04:41:53.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[24814]: mgrmap e29: y(active, since 1.28366s) 2024-03-26T04:41:53.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:53 smithi017 ceph-mon[24814]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-mon[25657]: [26/Mar/2024:04:41:51] ENGINE Bus STARTING 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-mon[25657]: [26/Mar/2024:04:41:51] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-mon[25657]: [26/Mar/2024:04:41:52] ENGINE Serving on http://172.21.15.17:8765 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-mon[25657]: [26/Mar/2024:04:41:52] ENGINE Bus STARTED 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-mon[25657]: mgrmap e29: y(active, since 1.28366s) 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-mon[25657]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.458+0000 7fb04be08200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:41:53.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.578+0000 7fb04be08200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:53.968 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.668+0000 7fb04be08200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:41:53.968 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.765+0000 7fb04be08200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:41:53.969 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:53.968+0000 7fb04be08200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:41:54.583 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:54 smithi017 ceph-mon[24814]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:54.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:54 smithi017 ceph-mon[24814]: mgrmap e30: y(active, since 2s) 2024-03-26T04:41:54.584 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:54 smithi017 ceph-mon[29581]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:54.584 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:54 smithi017 ceph-mon[29581]: mgrmap e30: y(active, since 2s) 2024-03-26T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:54 smithi129 ceph-mon[25657]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:54 smithi129 ceph-mon[25657]: mgrmap e30: y(active, since 2s) 2024-03-26T04:41:55.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:54.989+0000 7fb04be08200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:41:55.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:55.125+0000 7fb04be08200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:41:55.776 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:55.476+0000 7fb04be08200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:41:55.776 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:55.569+0000 7fb04be08200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:41:56.082 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:55.776+0000 7fb04be08200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:56.082 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:55.967+0000 7fb04be08200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:41:56.348 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:56 smithi017 ceph-mon[29581]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:56.349 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:56 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:56.349 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:56 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:56.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:56 smithi017 ceph-mon[24814]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:56.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:56 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:56.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:56 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:56.349 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:41:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:56 smithi129 ceph-mon[25657]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:56 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:56 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:56.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:56.303+0000 7fb04be08200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:41:56.834 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:56.580+0000 7fb04be08200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:41:56.835 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:56.834+0000 7fb04be08200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:41:57.080 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:41:57.080 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (57s) 1s ago 5m 16.9M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:41:57.080 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (24s) 0s ago 5m 42.8M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:41:57.080 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 unknown 1s ago 4m - - 2024-03-26T04:41:57.080 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283 running (102s) 0s ago 10m 74.4M - 19.0.0-1775-g5e70124b 5f21c9264136 7f62fb476af1 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (19s) 1s ago 11m 476M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (11m) 1s ago 11m 54.5M 2048M 17.2.0 e1d6a67b021e e512d90ccc03 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (10m) 0s ago 10m 45.8M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (10m) 1s ago 10m 42.7M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (66s) 1s ago 6m 10.0M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (38s) 0s ago 6m 9827k - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (9m) 1s ago 9m 53.0M 2737M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (9m) 1s ago 9m 55.0M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (9m) 1s ago 9m 51.7M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (8m) 1s ago 8m 53.0M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (8m) 0s ago 8m 55.4M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (8m) 0s ago 8m 51.4M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (7m) 0s ago 7m 53.8M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (7m) 0s ago 7m 53.4M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (41s) 0s ago 6m 54.0M - 2.33.4 514e6a882f6e 97c06de0d5df 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (5m) 1s ago 5m 71.7M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:41:57.081 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (5m) 0s ago 5m 75.5M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:41:57.082 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (4m) 1s ago 4m 72.2M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:41:57.082 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (4m) 0s ago 4m 72.8M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:41:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-mon[25657]: mgrmap e31: y(active, since 4s) 2024-03-26T04:41:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-mon[25657]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[24814]: mgrmap e31: y(active, since 4s) 2024-03-26T04:41:57.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[24814]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:57.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[29581]: mgrmap e31: y(active, since 4s) 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[29581]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:57 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:57.511 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:41:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:41:57.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:41:57.512 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:41:57.513 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/23 daemons upgraded", 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:41:57.861 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:41:58.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:57.691+0000 7fb04be08200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:41:58.267 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:41:58.588 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:58.263+0000 7fb04be08200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:41:58.588 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:58.366+0000 7fb04be08200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:41:58.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:58.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:58.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:58.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:58.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:58.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1515483617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:41:58.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/4103281708' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:41:58.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:58.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:58.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:58.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:58.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/1515483617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/4103281708' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1515483617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:41:58.647 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:58 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/4103281708' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:41:58.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:41:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:41:58.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:58.587+0000 7fb04be08200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:41:58.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:58] ENGINE Bus STARTING 2024-03-26T04:41:58.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: CherryPy Checker: 2024-03-26T04:41:58.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: The Application mounted at '' has an empty config. 2024-03-26T04:41:58.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: 2024-03-26T04:41:58.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:58] ENGINE Serving on http://:::9283 2024-03-26T04:41:58.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: [26/Mar/2024:04:41:58] ENGINE Bus STARTED 2024-03-26T04:41:59.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-mon[25657]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:59.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-mon[25657]: Standby manager daemon x started 2024-03-26T04:41:59.637 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:59.637 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:59.637 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:59.637 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[24814]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[24814]: Standby manager daemon x started 2024-03-26T04:41:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:59.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[29581]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:41:59.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[29581]: Standby manager daemon x started 2024-03-26T04:41:59.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:41:59.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:41:59.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:41:59.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:41:59 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1777550877' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:41:59.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:41:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[46783]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:41:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T04:42:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:00 smithi017 ceph-mon[24814]: mgrmap e32: y(active, since 8s), standbys: x 2024-03-26T04:42:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:00 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:42:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:00 smithi017 ceph-mon[24814]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:42:00.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:00 smithi017 ceph-mon[29581]: mgrmap e32: y(active, since 8s), standbys: x 2024-03-26T04:42:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:00 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:42:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:00 smithi017 ceph-mon[29581]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:42:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:00 smithi129 ceph-mon[25657]: mgrmap e32: y(active, since 8s), standbys: x 2024-03-26T04:42:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:00 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:42:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:00 smithi129 ceph-mon[25657]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T04:42:01.727 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:01.727 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:01.727 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:01.727 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:01.727 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:42:01.727 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:42:01.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:01.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:01.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:01.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:01.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:42:01.728 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:01 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:01 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:01 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:01 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:01 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:01 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:42:02.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:01 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:42:02.902 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.903 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:02 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:02.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:02 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:03.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[24814]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:42:03.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[24814]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:42:03.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[24814]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:42:03.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[24814]: from='client.? 172.21.15.17:0/3884893451' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:42:03.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[24814]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:42:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[29581]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:42:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[29581]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:42:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[29581]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:42:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/3884893451' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:42:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:03 smithi017 ceph-mon[29581]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:42:04.024 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:03 smithi129 ceph-mon[25657]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T04:42:04.024 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:03 smithi129 ceph-mon[25657]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:42:04.024 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:03 smithi129 ceph-mon[25657]: Reconfiguring daemon prometheus.a on smithi129 2024-03-26T04:42:04.024 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:03 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/3884893451' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:42:04.024 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:03 smithi129 ceph-mon[25657]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:42:05.395 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 systemd[1]: Stopping Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:05.395 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 bash[53643]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:42:05.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-mon[25657]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.475Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.475Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.475Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.475Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.475Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.475Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.476Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.476Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.476Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:42:05.743 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.476Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:42:05.744 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.476Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T04:42:05.744 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[50428]: ts=2024-03-26T04:42:05.476Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T04:42:05.744 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 podman[53656]: 2024-03-26 04:42:05.485346403 +0000 UTC m=+0.136551878 container died 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:05.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:05 smithi017 ceph-mon[24814]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:05.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:05 smithi017 ceph-mon[29581]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:06.109 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 podman[53656]: 2024-03-26 04:42:05.834199115 +0000 UTC m=+0.485404597 container cleanup 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:06.109 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 bash[53656]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:42:06.109 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:05 smithi129 bash[53682]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:42:06.360 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 podman[53670]: 2024-03-26 04:42:06.109924818 +0000 UTC m=+0.626023449 container remove 97c06de0d5df547892dfdd6282f9eba9fb797bedd72547a03bdac9f9ff142776 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:06.360 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T04:42:06.360 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 systemd[1]: Stopped Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:06.360 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 systemd[1]: Starting Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:06.360 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 podman[53755]: 2024-03-26 04:42:06.251422939 +0000 UTC m=+0.019714614 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T04:42:06.360 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 podman[53755]: 2024-03-26 04:42:06.361077281 +0000 UTC m=+0.129368932 container create 807223168a4309fa7220a2802c457b599564bc5f1cfe859f2ac0e1ea6566b1e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:06.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 podman[53755]: 2024-03-26 04:42:06.571954893 +0000 UTC m=+0.340246553 container init 807223168a4309fa7220a2802c457b599564bc5f1cfe859f2ac0e1ea6566b1e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:06.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 podman[53755]: 2024-03-26 04:42:06.575217446 +0000 UTC m=+0.343509106 container start 807223168a4309fa7220a2802c457b599564bc5f1cfe859f2ac0e1ea6566b1e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.597Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.597Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.597Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.597Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi129 (none))" 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.597Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.597Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.599Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.599Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.601Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.603Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.603Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.478µs 2024-03-26T04:42:06.646 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.603Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:42:07.052 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 bash[53755]: 807223168a4309fa7220a2802c457b599564bc5f1cfe859f2ac0e1ea6566b1e7 2024-03-26T04:42:07.052 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 systemd[1]: Started Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:07.052 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:06.805Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-26T04:42:07.052 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.052Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.088Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.089Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.089Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.051µs wal_replay_duration=485.635779ms total_replay_duration=485.69736ms 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.092Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.092Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.092Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:42:07.372 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.111Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.007707ms db_storage=772ns remote_storage=1.308µs web_handler=640ns query_engine=654ns scrape=371.66µs scrape_sd=78.193µs notify=16.982µs notify_sd=16.384µs rules=18.300386ms 2024-03-26T04:42:07.373 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:07.111Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T04:42:07.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:07.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:07.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.626 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.626 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:42:07.626 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:07 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:07.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:07.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:07.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:42:07.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:07 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:08 smithi129 ceph-mon[25657]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:08 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:08 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:08 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:42:08.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:08 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:08.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[24814]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[29581]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:42:08.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:08 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:09.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:09 smithi129 ceph-mon[25657]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:09.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:09 smithi129 ceph-mon[25657]: Upgrade: Updating mgr.x 2024-03-26T04:42:09.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:09 smithi129 ceph-mon[25657]: Deploying daemon mgr.x on smithi129 2024-03-26T04:42:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:09 smithi017 ceph-mon[24814]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:09 smithi017 ceph-mon[24814]: Upgrade: Updating mgr.x 2024-03-26T04:42:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:09 smithi017 ceph-mon[24814]: Deploying daemon mgr.x on smithi129 2024-03-26T04:42:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:09 smithi017 ceph-mon[29581]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:09 smithi017 ceph-mon[29581]: Upgrade: Updating mgr.x 2024-03-26T04:42:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:09 smithi017 ceph-mon[29581]: Deploying daemon mgr.x on smithi129 2024-03-26T04:42:10.341 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:10 smithi129 systemd[1]: Stopping Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:10.611 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:10 smithi129 podman[54100]: 2024-03-26 04:42:10.341467372 +0000 UTC m=+0.145526891 container died 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-26T04:42:10.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:10 smithi129 podman[54100]: 2024-03-26 04:42:10.685599786 +0000 UTC m=+0.489659311 container cleanup 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:42:10.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:10 smithi129 bash[54100]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:42:11.285 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:10 smithi129 podman[54114]: 2024-03-26 04:42:10.978617968 +0000 UTC m=+0.636606047 container remove 7f62fb476af1e8d1a89e0aad2e1789621acc408e3acd30420f98ec7192e17606 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:42:11.285 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:10 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:42:11.285 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-26T04:42:11.286 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 systemd[1]: Stopped Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:11.286 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service: Consumed 27.123s CPU time. 2024-03-26T04:42:11.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:11 smithi129 ceph-mon[25657]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-26T04:42:11.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 systemd[1]: Starting Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:11.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 podman[54248]: 2024-03-26 04:42:11.403580998 +0000 UTC m=+0.018013766 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:42:11.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 podman[54248]: 2024-03-26 04:42:11.530239018 +0000 UTC m=+0.144671785 container create 7a621834387fe27744a663186124fae2b7aa0f06fef8187806c56c21e38c94c0 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T04:42:11.757 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:11 smithi017 ceph-mon[29581]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-26T04:42:11.758 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:11 smithi017 ceph-mon[24814]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-26T04:42:12.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 podman[54248]: 2024-03-26 04:42:11.758268813 +0000 UTC m=+0.372701581 container init 7a621834387fe27744a663186124fae2b7aa0f06fef8187806c56c21e38c94c0 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:42:12.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 podman[54248]: 2024-03-26 04:42:11.760868448 +0000 UTC m=+0.375301217 container start 7a621834387fe27744a663186124fae2b7aa0f06fef8187806c56c21e38c94c0 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T04:42:12.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 ceph-mgr[54267]: -- 172.21.15.129:0/1595957882 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e95837e1a0 con 0x55e958354c00 2024-03-26T04:42:12.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 bash[54248]: 7a621834387fe27744a663186124fae2b7aa0f06fef8187806c56c21e38c94c0 2024-03-26T04:42:12.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:11 smithi129 systemd[1]: Started Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:12.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:12 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:12.345+0000 7fbd14210200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:42:12.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:12 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:12.454+0000 7fbd14210200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:42:12.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:12 smithi129 ceph-mon[25657]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-26T04:42:12.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:12 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:12.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:12 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:12.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:12 smithi129 ceph-mon[25657]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:13.188 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[24814]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-26T04:42:13.188 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:13.188 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:13.188 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[24814]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:13.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[29581]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 594 B/s rd, 0 op/s 2024-03-26T04:42:13.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:13.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:13.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:12 smithi017 ceph-mon[29581]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:13.387 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.132+0000 7fbd14210200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:42:13.387 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.238+0000 7fbd14210200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:42:13.640 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.386+0000 7fbd14210200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:42:13.640 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.484+0000 7fbd14210200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:42:13.640 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.573+0000 7fbd14210200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:42:13.895 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.848+0000 7fbd14210200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:42:14.358 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:13.963+0000 7fbd14210200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:42:14.358 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:14.060+0000 7fbd14210200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:42:14.358 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:14.171+0000 7fbd14210200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:42:14.358 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:14 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:14.357+0000 7fbd14210200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:42:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:14 smithi017 ceph-mon[24814]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T04:42:15.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:14 smithi017 ceph-mon[29581]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T04:42:15.227 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:14 smithi129 ceph-mon[25657]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T04:42:15.518 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:15.380+0000 7fbd14210200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:42:15.518 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:15.517+0000 7fbd14210200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:42:15.869 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:15.869+0000 7fbd14210200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:42:16.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:15 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:15.962+0000 7fbd14210200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:42:16.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:16.164+0000 7fbd14210200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:42:16.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:16.370+0000 7fbd14210200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:42:16.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:42:16] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.33.4" 2024-03-26T04:42:16.954 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:16.678+0000 7fbd14210200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:42:17.208 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:16 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:16.954+0000 7fbd14210200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:42:17.208 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:17 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:17.207+0000 7fbd14210200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:42:17.507 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:17 smithi129 ceph-mon[25657]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:17.507 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:17 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:17.507 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:17 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:17 smithi017 ceph-mon[24814]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:17 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:17.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:17 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:17.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:17 smithi017 ceph-mon[29581]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:17.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:17 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:17.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:17 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:18.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:18.092+0000 7fbd14210200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:42:18.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:18 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:18.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:18 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:18.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:18 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:18.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:18 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:18.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:18 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:18.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:18 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:19.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:18.686+0000 7fbd14210200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:42:19.013 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:18.787+0000 7fbd14210200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:42:19.013 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:19.012+0000 7fbd14210200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:42:19.013 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:42:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: [26/Mar/2024:04:42:19] ENGINE Bus STARTING 2024-03-26T04:42:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: CherryPy Checker: 2024-03-26T04:42:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: The Application mounted at '' has an empty config. 2024-03-26T04:42:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: [26/Mar/2024:04:42:19] ENGINE Serving on http://:::9283 2024-03-26T04:42:19.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: [26/Mar/2024:04:42:19] ENGINE Bus STARTED 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: Standby manager daemon x restarted 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: Standby manager daemon x started 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:42:19.716 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:19 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:42:19.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:19.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: Standby manager daemon x restarted 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: Standby manager daemon x started 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[24814]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: Standby manager daemon x restarted 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: Standby manager daemon x started 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:42:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:19 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/2622109830' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:42:20.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:20 smithi129 ceph-mon[25657]: mgrmap e33: y(active, since 28s), standbys: x 2024-03-26T04:42:20.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:20 smithi017 ceph-mon[24814]: mgrmap e33: y(active, since 28s), standbys: x 2024-03-26T04:42:20.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:20 smithi017 ceph-mon[29581]: mgrmap e33: y(active, since 28s), standbys: x 2024-03-26T04:42:21.803 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:21.803 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.803 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:42:21.804 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T04:42:21.805 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-26T04:42:21.806 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:21 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:21.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-26T04:42:21.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:21 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[24814]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-03-26T04:42:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[24814]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[24814]: Upgrade: Setting container_image for all mgr 2024-03-26T04:42:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[24814]: Upgrade: It appears safe to stop mon.a 2024-03-26T04:42:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[29581]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-03-26T04:42:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[29581]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[29581]: Upgrade: Setting container_image for all mgr 2024-03-26T04:42:22.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:22 smithi017 ceph-mon[29581]: Upgrade: It appears safe to stop mon.a 2024-03-26T04:42:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:22 smithi129 ceph-mon[25657]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-03-26T04:42:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:22 smithi129 ceph-mon[25657]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:22 smithi129 ceph-mon[25657]: Upgrade: Setting container_image for all mgr 2024-03-26T04:42:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:22 smithi129 ceph-mon[25657]: Upgrade: It appears safe to stop mon.a 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[24814]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[24814]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[24814]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[29581]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:42:23.696 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:23 smithi017 ceph-mon[29581]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:23 smithi129 ceph-mon[25657]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:23 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:23 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:42:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:23 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:42:23.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:23 smithi129 ceph-mon[25657]: from='mgr.15324 172.21.15.17:0/293913222' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:24.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:24 smithi017 ceph-mon[24814]: Upgrade: Updating mon.a 2024-03-26T04:42:24.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:24 smithi017 ceph-mon[24814]: Deploying daemon mon.a on smithi017 2024-03-26T04:42:24.578 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:24 smithi017 ceph-mon[24814]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:24.578 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:24 smithi017 ceph-mon[29581]: Upgrade: Updating mon.a 2024-03-26T04:42:24.578 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:24 smithi017 ceph-mon[29581]: Deploying daemon mon.a on smithi017 2024-03-26T04:42:24.578 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:24 smithi017 ceph-mon[29581]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:24 smithi129 ceph-mon[25657]: Upgrade: Updating mon.a 2024-03-26T04:42:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:24 smithi129 ceph-mon[25657]: Deploying daemon mon.a on smithi017 2024-03-26T04:42:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:24 smithi129 ceph-mon[25657]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:25.117 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:24 smithi017 systemd[1]: Stopping Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a[24810]: 2024-03-26T04:42:25.115+0000 7f46fa04d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a[24810]: 2024-03-26T04:42:25.115+0000 7f46fa04d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T04:42:25.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 podman[64420]: 2024-03-26 04:42:25.197865342 +0000 UTC m=+0.193937966 container died e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.openshift.expose-services=, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, vendor=Red Hat, Inc., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, distribution-scope=public, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T04:42:25.783 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 podman[64420]: 2024-03-26 04:42:25.574276654 +0000 UTC m=+0.570349274 container cleanup e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, version=8, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, release=754) 2024-03-26T04:42:25.783 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 bash[64420]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:42:26.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 podman[64434]: 2024-03-26 04:42:25.783850169 +0000 UTC m=+0.586955317 container remove e512d90ccc03fd1061a998b2c24d98c75a3b63eabf8ed92d89f8e25e8b8b0ea0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_CLEAN=True, io.buildah.version=1.19.8, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-26T04:42:26.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:42:26.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 systemd[1]: Stopped Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:26.185 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:25 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service: Consumed 13.846s CPU time. 2024-03-26T04:42:26.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:42:26] "GET /metrics HTTP/1.1" 200 38238 "" "Prometheus/2.33.4" 2024-03-26T04:42:26.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 systemd[1]: Starting Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 podman[64566]: 2024-03-26 04:42:26.392939897 +0000 UTC m=+0.110262920 container create c90cfaf45cc1388c245e63ba39466e875010a898df1edee57130cf18024e6b3d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:42:26.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 podman[64566]: 2024-03-26 04:42:26.300582799 +0000 UTC m=+0.017905851 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 podman[64566]: 2024-03-26 04:42:26.620996406 +0000 UTC m=+0.338319439 container init c90cfaf45cc1388c245e63ba39466e875010a898df1edee57130cf18024e6b3d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 podman[64566]: 2024-03-26 04:42:26.62420337 +0000 UTC m=+0.341526406 container start c90cfaf45cc1388c245e63ba39466e875010a898df1edee57130cf18024e6b3d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: pidfile_write: ignore empty --pid-file 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: load: jerasure load: lrc 2024-03-26T04:42:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Git sha 0 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: DB SUMMARY 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: DB Session ID: I9JT1KKFW7337OLGJZDN 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3320138 ; 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:42:26.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:42:26.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:42:26.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.env: 0x55c061da4ae0 2024-03-26T04:42:26.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:42:26.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.info_log: 0x55c0637f3920 2024-03-26T04:42:26.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:42:26.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.statistics: (nil) 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.use_fsync: 0 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:42:26.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.db_log_dir: 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.wal_dir: 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:42:26.963 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.write_buffer_manager: 0x55c0637c6820 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.unordered_write: 0 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:42:26.964 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.row_cache: None 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.wal_filter: None 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.wal_compression: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:42:26.965 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:42:26.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_open_files: -1 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Compression algorithms supported: 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kZSTD supported: 0 2024-03-26T04:42:26.967 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.merge_operator: 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_filter: None 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:42:26.968 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c0637f3a20) 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 4 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x55c0637f7090 2024-03-26T04:42:26.969 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-03-26T04:42:26.970 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: bloomfilter 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 5 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: max_auto_readahead_size: 262144 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: prepopulate_block_cache: 0 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression: NoCompression 2024-03-26T04:42:26.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.num_levels: 7 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:42:26.972 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:42:26.973 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:42:26.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:42:26.975 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.ttl: 2592000 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:42:26.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12140, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:42:26.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 927481eb-d35f-4893-a7bb-b84761848673 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428146653510, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428146665303, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3054407, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12141, "largest_seqno": 13508, "table_properties": {"data_size": 3047705, "index_size": 4147, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 14377, "raw_average_key_size": 23, "raw_value_size": 3034408, "raw_average_value_size": 5023, "num_data_blocks": 189, "num_entries": 604, "num_filter_entries": 604, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711428146, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "927481eb-d35f-4893-a7bb-b84761848673", "db_session_id": "I9JT1KKFW7337OLGJZDN", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428146665414, "job": 1, "event": "recovery_finished"} 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c0638e8000 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: DB pointer 0x55c0638d2000 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-03-26T04:42:26.978 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 1/0 2.91 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: L6 1/0 7.95 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 10.86 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-26T04:42:26.979 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 593.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 593.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:42:26.980 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 201.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 201.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache BinnedLRUCache@0x55c0637f7090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.7e-05 secs_since: 0 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???) e3 preinit fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).mds e1 new map 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T04:42:26.981 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 ceph-mon[64584]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 bash[64566]: c90cfaf45cc1388c245e63ba39466e875010a898df1edee57130cf18024e6b3d 2024-03-26T04:42:26.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:26 smithi017 systemd[1]: Started Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:28.640 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:42:29.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (89s) 33s ago 6m 16.9M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (57s) 12s ago 6m 43.0M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 unknown 33s ago 5m - - 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (17s) 12s ago 10m 63.6M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (51s) 33s ago 12m 476M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 starting - - - 2048M 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (11m) 12s ago 11m 47.3M 2048M 17.2.0 e1d6a67b021e 685e2a3e6efc 2024-03-26T04:42:29.310 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (11m) 33s ago 11m 42.7M 2048M 17.2.0 e1d6a67b021e 130e1901a8fc 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (98s) 33s ago 7m 10.0M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (70s) 12s ago 7m 11.3M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (10m) 33s ago 10m 53.0M 2737M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (10m) 33s ago 10m 55.0M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (9m) 33s ago 9m 51.7M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (9m) 33s ago 9m 53.0M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (8m) 12s ago 8m 55.7M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (8m) 12s ago 8m 51.7M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (8m) 12s ago 8m 54.2M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (7m) 12s ago 7m 53.9M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (23s) 12s ago 7m 44.6M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (5m) 33s ago 5m 71.7M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (5m) 12s ago 5m 76.0M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (5m) 33s ago 5m 72.2M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:42:29.311 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (5m) 12s ago 5m 73.4M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:42:29.731 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:42:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:42:29.731 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:42:29.732 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-26T04:42:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:42:29.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:42:29.733 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:42:30.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: ignoring --setuser ceph since I am not root 2024-03-26T04:42:30.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: ignoring --setgroup ceph since I am not root 2024-03-26T04:42:30.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mgr[54267]: -- 172.21.15.129:0/2799264253 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5582201001a0 con 0x5582200d6c00 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: mon.a calling monitor election 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 549 B/s rd, 0 op/s 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: from='client.25142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: fsmap 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: mgrmap e33: y(active, since 38s), standbys: x 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T04:42:30.013 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:42:30.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:42:30.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:30.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: mgrmap e34: y(active, since 38s), standbys: x 2024-03-26T04:42:30.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/2548123609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:30.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:29 smithi129 ceph-mon[25657]: from='mgr.15324 ' entity='' 2024-03-26T04:42:30.171 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ignoring --setuser ceph since I am not root 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ignoring --setgroup ceph since I am not root 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mgr[60299]: -- 172.21.15.17:0/3532248105 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e835821a0 con 0x557e83559000 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: mon.a calling monitor election 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 549 B/s rd, 0 op/s 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: from='client.25142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:30.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: fsmap 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: mgrmap e33: y(active, since 38s), standbys: x 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: mgrmap e34: y(active, since 38s), standbys: x 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/2548123609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:30.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[29581]: from='mgr.15324 ' entity='' 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: mon.a calling monitor election 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 549 B/s rd, 0 op/s 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: from='client.25142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: fsmap 2024-03-26T04:42:30.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: mgrmap e33: y(active, since 38s), standbys: x 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: from='mgr.15324 ' entity='mgr.y' 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: mgrmap e34: y(active, since 38s), standbys: x 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2548123609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:30.175 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:29 smithi017 ceph-mon[64584]: from='mgr.15324 ' entity='' 2024-03-26T04:42:30.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:30.316+0000 7f87e83ef200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:42:30.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:30 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:30.412+0000 7f87e83ef200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:42:30.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:30.312+0000 7fc82769d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:42:30.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:30.414+0000 7fc82769d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:42:31.321 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.084+0000 7fc82769d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:42:31.321 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.178+0000 7fc82769d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:42:31.321 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.320+0000 7fc82769d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:42:31.416 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.076+0000 7f87e83ef200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:42:31.416 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.173+0000 7f87e83ef200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:42:31.416 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.317+0000 7f87e83ef200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:42:31.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.418+0000 7fc82769d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:42:31.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.502+0000 7fc82769d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:42:31.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.415+0000 7f87e83ef200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:42:31.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.501+0000 7f87e83ef200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:42:31.974 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-mon[64584]: mgrmap e35: y(active, since 39s), standbys: x 2024-03-26T04:42:31.975 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-mon[29581]: mgrmap e35: y(active, since 39s), standbys: x 2024-03-26T04:42:31.975 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.772+0000 7f87e83ef200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:42:31.975 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.885+0000 7f87e83ef200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:42:32.053 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-mon[25657]: mgrmap e35: y(active, since 39s), standbys: x 2024-03-26T04:42:32.053 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.766+0000 7fc82769d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:42:32.053 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.873+0000 7fc82769d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:42:32.053 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:31.959+0000 7fc82769d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:42:32.054 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:32.053+0000 7fc82769d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:42:32.251 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:31 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:31.974+0000 7f87e83ef200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:42:32.251 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:32 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:32.071+0000 7f87e83ef200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:42:32.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:32.227+0000 7fc82769d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:42:32.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:32 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:32.251+0000 7f87e83ef200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:42:33.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:33.190+0000 7fc82769d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:42:33.645 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:33.318+0000 7fc82769d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:42:33.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:33.238+0000 7f87e83ef200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:42:33.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:33.373+0000 7f87e83ef200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:42:33.940 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:33.650+0000 7fc82769d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:42:33.940 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:33.740+0000 7fc82769d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:42:34.012 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:33.715+0000 7f87e83ef200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:42:34.012 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:33.808+0000 7f87e83ef200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:42:34.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:33.939+0000 7fc82769d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:42:34.395 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:34.127+0000 7fc82769d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:42:34.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:34.012+0000 7f87e83ef200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:42:34.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:34.204+0000 7f87e83ef200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:42:34.673 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:34.413+0000 7fc82769d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:42:34.764 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:34.500+0000 7f87e83ef200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:42:35.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:34.673+0000 7fc82769d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:42:35.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:34.911+0000 7fc82769d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:42:35.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:34.764+0000 7f87e83ef200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:42:35.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:35.010+0000 7f87e83ef200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:42:36.145 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:35.721+0000 7fc82769d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:42:36.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:35 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:35.848+0000 7f87e83ef200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:42:36.614 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:36.295+0000 7fc82769d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:42:36.614 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:36.393+0000 7fc82769d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:42:36.614 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:36.614+0000 7fc82769d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:42:36.688 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:36.434+0000 7f87e83ef200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:42:36.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:36.537+0000 7f87e83ef200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:42:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-mon[25657]: Standby manager daemon x restarted 2024-03-26T04:42:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-mon[25657]: Standby manager daemon x started 2024-03-26T04:42:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:42:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:42:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:42:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-mon[25657]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b[25652]: 2024-03-26T04:42:36.782+0000 7f4cea614700 -1 mon.b@2(peon).osd e106 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: [26/Mar/2024:04:42:36] ENGINE Bus STARTING 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: CherryPy Checker: 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: The Application mounted at '' has an empty config. 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: [26/Mar/2024:04:42:36] ENGINE Serving on http://:::9283 2024-03-26T04:42:36.896 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:42:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x[54263]: [26/Mar/2024:04:42:36] ENGINE Bus STARTED 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:36.766+0000 7f87e83ef200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:42:36] ENGINE Bus STARTING 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[29581]: Standby manager daemon x restarted 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[29581]: Standby manager daemon x started 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[29581]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:42:36.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c[29577]: 2024-03-26T04:42:36.782+0000 7f1e91f40700 -1 mon.c@1(peon).osd e106 update_from_paxos full map CRC mismatch, resetting to canonical 2024-03-26T04:42:36.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[64584]: Standby manager daemon x restarted 2024-03-26T04:42:36.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[64584]: Standby manager daemon x started 2024-03-26T04:42:36.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[64584]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:42:36.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[64584]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:42:36.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[64584]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:42:36.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:36 smithi017 ceph-mon[64584]: from='mgr.? 172.21.15.129:0/1138784901' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:42:37.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: CherryPy Checker: 2024-03-26T04:42:37.205 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: The Application mounted at '' has an empty config. 2024-03-26T04:42:37.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:37.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:42:37] ENGINE Serving on http://:::9283 2024-03-26T04:42:37.206 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:42:37] ENGINE Bus STARTED 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/23 daemons upgraded", 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:42:37.798 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:42:37.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:37.798+0000 7f879e59b700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: mgrmap e36: y(active, since 45s), standbys: x 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: Active manager daemon y restarted 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: Activating manager daemon y 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: mgrmap e37: y(active, starting, since 0.00604626s), standbys: x 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: failed to encode map e106 with expected crc 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:42:37.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: failed to encode map e106 with expected crc 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:42:37.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: Manager daemon y is now available 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: mgrmap e36: y(active, since 45s), standbys: x 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: Active manager daemon y restarted 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: Activating manager daemon y 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: mgrmap e37: y(active, starting, since 0.00604626s), standbys: x 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:42:37.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: failed to encode map e106 with expected crc 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: failed to encode map e106 with expected crc 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:42:37.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:42:37.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:42:37.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: Manager daemon y is now available 2024-03-26T04:42:37.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:37.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:37.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:42:37.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: mgrmap e36: y(active, since 45s), standbys: x 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: Active manager daemon y restarted 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: Activating manager daemon y 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: mgrmap e37: y(active, starting, since 0.00604626s), standbys: x 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: failed to encode map e106 with expected crc 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:42:38.057 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: failed to encode map e106 with expected crc 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:42:38.058 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: Manager daemon y is now available 2024-03-26T04:42:38.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:38.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:38.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:42:38.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:37 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:42:38.217 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T04:42:38.217 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:42:38.217 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:42:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:38 smithi129 ceph-mon[25657]: [26/Mar/2024:04:42:37] ENGINE Bus STARTING 2024-03-26T04:42:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:38 smithi129 ceph-mon[25657]: [26/Mar/2024:04:42:37] ENGINE Serving on http://172.21.15.17:8765 2024-03-26T04:42:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:38 smithi129 ceph-mon[25657]: [26/Mar/2024:04:42:37] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:42:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:38 smithi129 ceph-mon[25657]: [26/Mar/2024:04:42:37] ENGINE Bus STARTED 2024-03-26T04:42:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:38 smithi129 ceph-mon[25657]: mgrmap e38: y(active, since 1.01455s), standbys: x 2024-03-26T04:42:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:38 smithi129 ceph-mon[25657]: from='client.? 172.21.15.17:0/1955414124' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:42:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[64584]: [26/Mar/2024:04:42:37] ENGINE Bus STARTING 2024-03-26T04:42:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[64584]: [26/Mar/2024:04:42:37] ENGINE Serving on http://172.21.15.17:8765 2024-03-26T04:42:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[64584]: [26/Mar/2024:04:42:37] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:42:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[64584]: [26/Mar/2024:04:42:37] ENGINE Bus STARTED 2024-03-26T04:42:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[64584]: mgrmap e38: y(active, since 1.01455s), standbys: x 2024-03-26T04:42:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1955414124' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:42:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[29581]: [26/Mar/2024:04:42:37] ENGINE Bus STARTING 2024-03-26T04:42:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[29581]: [26/Mar/2024:04:42:37] ENGINE Serving on http://172.21.15.17:8765 2024-03-26T04:42:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[29581]: [26/Mar/2024:04:42:37] ENGINE Serving on https://172.21.15.17:7150 2024-03-26T04:42:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[29581]: [26/Mar/2024:04:42:37] ENGINE Bus STARTED 2024-03-26T04:42:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[29581]: mgrmap e38: y(active, since 1.01455s), standbys: x 2024-03-26T04:42:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:38 smithi017 ceph-mon[29581]: from='client.? 172.21.15.17:0/1955414124' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[64584]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[64584]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[64584]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[64584]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[29581]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[29581]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[29581]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:39.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[29581]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T04:42:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[29581]: Cluster is now healthy 2024-03-26T04:42:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:39 smithi017 ceph-mon[29581]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:39 smithi129 ceph-mon[25657]: from='client.34113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:42:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:39 smithi129 ceph-mon[25657]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:39 smithi129 ceph-mon[25657]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:39 smithi129 ceph-mon[25657]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T04:42:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:39 smithi129 ceph-mon[25657]: Cluster is now healthy 2024-03-26T04:42:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:39 smithi129 ceph-mon[25657]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:40.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:40 smithi017 ceph-mon[64584]: mgrmap e39: y(active, since 3s), standbys: x 2024-03-26T04:42:40.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:40 smithi017 ceph-mon[29581]: mgrmap e39: y(active, since 3s), standbys: x 2024-03-26T04:42:41.108 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:40 smithi129 ceph-mon[25657]: mgrmap e39: y(active, since 3s), standbys: x 2024-03-26T04:42:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:42 smithi129 ceph-mon[25657]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:42 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:42 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:42 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:42 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:42 smithi129 ceph-mon[25657]: mgrmap e40: y(active, since 5s), standbys: x 2024-03-26T04:42:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[64584]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[64584]: mgrmap e40: y(active, since 5s), standbys: x 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[29581]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:42 smithi017 ceph-mon[29581]: mgrmap e40: y(active, since 5s), standbys: x 2024-03-26T04:42:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:43 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:43 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:43 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:43 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:43.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:43 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:44 smithi129 ceph-mon[25657]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:45.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:44 smithi017 ceph-mon[64584]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:45.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:44 smithi017 ceph-mon[29581]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:42:46.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:42:46] "GET /metrics HTTP/1.1" 200 35449 "" "Prometheus/2.33.4" 2024-03-26T04:42:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:46 smithi129 ceph-mon[25657]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:42:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:46 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:46 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:46 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:46 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:47.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[64584]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:42:47.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:47.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:47.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:47.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[29581]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:42:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:47.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:46 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:48.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:42:48.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:42:48.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.048 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:48.048 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:48.048 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:42:48.048 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:47 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-26T04:42:48.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.conf 2024-03-26T04:42:48.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-26T04:42:48.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:48.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:42:48.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:48.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:42:48.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:47 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-26T04:42:48.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:42:48.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:48.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: Upgrade: It appears safe to stop mon.c 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[29581]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:48.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: Upgrade: It appears safe to stop mon.c 2024-03-26T04:42:48.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:48.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:42:48.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:42:48.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:49.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:42:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:42:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: Updating smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: Updating smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:42:49.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: Upgrade: It appears safe to stop mon.c 2024-03-26T04:42:49.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:49.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:42:49.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:42:49.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:48 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:49.839 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:49 smithi017 ceph-mon[29581]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:49.839 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:49 smithi017 ceph-mon[29581]: Upgrade: Updating mon.c 2024-03-26T04:42:49.839 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:49 smithi017 ceph-mon[29581]: Deploying daemon mon.c on smithi017 2024-03-26T04:42:49.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:49 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:49.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:49 smithi017 ceph-mon[64584]: Upgrade: Updating mon.c 2024-03-26T04:42:49.839 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:49 smithi017 ceph-mon[64584]: Deploying daemon mon.c on smithi017 2024-03-26T04:42:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:49 smithi129 ceph-mon[25657]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:49 smithi129 ceph-mon[25657]: Upgrade: Updating mon.c 2024-03-26T04:42:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:49 smithi129 ceph-mon[25657]: Deploying daemon mon.c on smithi017 2024-03-26T04:42:50.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:50 smithi017 systemd[1]: Stopping Ceph mon.c for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:50.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:50 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c[29577]: 2024-03-26T04:42:50.692+0000 7f1e9774b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:42:50.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:50 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c[29577]: 2024-03-26T04:42:50.692+0000 7f1e9774b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:42:50.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:50 smithi017 podman[66978]: 2024-03-26 04:42:50.723596578 +0000 UTC m=+0.151510072 container died 130e1901a8fcfc3ed7689418a83bc308373f9cb1bc38560fc5bcfc3608c52bb5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, RELEASE=HEAD, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_CLEAN=True, vcs-type=git, build-date=2022-05-03T08:36:31.336870, ceph=True, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-26T04:42:51.359 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 podman[66978]: 2024-03-26 04:42:51.026043799 +0000 UTC m=+0.453957295 container cleanup 130e1901a8fcfc3ed7689418a83bc308373f9cb1bc38560fc5bcfc3608c52bb5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T04:42:51.359 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 bash[66978]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c 2024-03-26T04:42:51.359 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 podman[66993]: 2024-03-26 04:42:51.244076959 +0000 UTC m=+0.515147141 container remove 130e1901a8fcfc3ed7689418a83bc308373f9cb1bc38560fc5bcfc3608c52bb5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, ceph=True, distribution-scope=public, vcs-type=git, version=8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, architecture=x86_64, name=centos-stream) 2024-03-26T04:42:51.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-26T04:42:51.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 systemd[1]: Stopped Ceph mon.c for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:51.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.c.service: Consumed 8.057s CPU time. 2024-03-26T04:42:51.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 systemd[1]: Starting Ceph mon.c for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:42:52.057 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 podman[67124]: 2024-03-26 04:42:51.698078589 +0000 UTC m=+0.018046265 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:42:52.057 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:51 smithi017 podman[67124]: 2024-03-26 04:42:51.795449146 +0000 UTC m=+0.115416825 container create 2c7ea6faebf3d19832fa3075a1d7847bd31007348ace89f1de6e0dba8e33bf06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T04:42:52.057 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 podman[67124]: 2024-03-26 04:42:52.031115995 +0000 UTC m=+0.351083671 container init 2c7ea6faebf3d19832fa3075a1d7847bd31007348ace89f1de6e0dba8e33bf06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124) 2024-03-26T04:42:52.057 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 podman[67124]: 2024-03-26 04:42:52.034190157 +0000 UTC m=+0.354157833 container start 2c7ea6faebf3d19832fa3075a1d7847bd31007348ace89f1de6e0dba8e33bf06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T04:42:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[64584]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: pidfile_write: ignore empty --pid-file 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: load: jerasure load: lrc 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Git sha 0 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: DB SUMMARY 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: DB Session ID: XPMV6XAI8V3JMZOGF0KJ 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:42:52.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 902642 ; 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.env: 0x5587b601bae0 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.info_log: 0x5587b7d09920 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:42:52.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.statistics: (nil) 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.use_fsync: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:42:52.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.db_log_dir: 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.wal_dir: 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.write_buffer_manager: 0x5587b7cdc820 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:42:52.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.unordered_write: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.row_cache: None 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.wal_filter: None 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:42:52.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.wal_compression: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:42:52.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:42:52.463 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_open_files: -1 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Compression algorithms supported: 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kZSTD supported: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:42:52.464 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.merge_operator: 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_filter: None 2024-03-26T04:42:52.465 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5587b7d09a20) 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_type: 0 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_index_type: 0 2024-03-26T04:42:52.466 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_shortening: 1 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: checksum: 4 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: no_block_cache: 0 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache: 0x5587b7d0d090 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_options: 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: capacity : 536870912 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_shard_bits : 4 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: strict_capacity_limit : 0 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_compressed: (nil) 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: persistent_cache: (nil) 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size: 4096 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size_deviation: 10 2024-03-26T04:42:52.467 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_restart_interval: 16 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_block_restart_interval: 1 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: metadata_block_size: 4096 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: partition_filters: 0 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: use_delta_encoding: 1 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: filter_policy: bloomfilter 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: whole_key_filtering: 1 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: verify_compression: 0 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: format_version: 5 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_index_compression: 1 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_align: 0 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: max_auto_readahead_size: 262144 2024-03-26T04:42:52.468 INFO:journalctl@ceph.mon.c.smithi017.stdout: prepopulate_block_cache: 0 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression: NoCompression 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:42:52.469 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.num_levels: 7 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:42:52.470 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:42:52.471 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:42:52.472 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:42:52.473 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.ttl: 2592000 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:42:52.474 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14882, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-26T04:42:52.475 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b3068b07-e3e1-431d-b0c1-9fd2b2c0b31e 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428172063388, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428172066540, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 558734, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14887, "largest_seqno": 15115, "table_properties": {"data_size": 556634, "index_size": 768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2708, "raw_average_key_size": 24, "raw_value_size": 554045, "raw_average_value_size": 5036, "num_data_blocks": 34, "num_entries": 110, "num_filter_entries": 110, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711428172, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b3068b07-e3e1-431d-b0c1-9fd2b2c0b31e", "db_session_id": "XPMV6XAI8V3JMZOGF0KJ", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428172066634, "job": 1, "event": "recovery_finished"} 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5587b7dfe000 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: DB pointer 0x5587b7de8000 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** DB Stats ** 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:42:52.476 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: L0 1/0 545.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 297.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: L6 1/0 11.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Sum 2/0 11.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 297.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 297.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:42:52.477 INFO:journalctl@ceph.mon.c.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 297.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative compaction: 0.00 GB write, 101.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval compaction: 0.00 GB write, 101.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Block cache BinnedLRUCache@0x5587b7d0d090#2 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-26T04:42:52.478 INFO:journalctl@ceph.mon.c.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.83 KB,0.000157952%) Misc(1,0.00 KB,0%) 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: starting mon.c rank 1 at public addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] at bind addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???) e3 preinit fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).mds e1 new map 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).mds e1 print_map 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: e1 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: legacy client fscid: -1 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout: No filesystems configured 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:42:52.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:42:52.480 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-03-26T04:42:52.480 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 bash[67124]: 2c7ea6faebf3d19832fa3075a1d7847bd31007348ace89f1de6e0dba8e33bf06 2024-03-26T04:42:52.480 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 systemd[1]: Started Ceph mon.c for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:42:52.480 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:52 smithi017 ceph-mon[67142]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:52.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:52 smithi129 ceph-mon[25657]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: mon.c calling monitor election 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: mon.a calling monitor election 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: fsmap 2024-03-26T04:42:53.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: mgrmap e40: y(active, since 15s), standbys: x 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: overall HEALTH_OK 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: mon.c calling monitor election 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: mon.a calling monitor election 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:42:53.318 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: fsmap 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: mgrmap e40: y(active, since 15s), standbys: x 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: overall HEALTH_OK 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:53.319 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:53 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: mon.c calling monitor election 2024-03-26T04:42:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: mon.a calling monitor election 2024-03-26T04:42:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:42:53.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: fsmap 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: mgrmap e40: y(active, since 15s), standbys: x 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: overall HEALTH_OK 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:53.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:53 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:54 smithi129 ceph-mon[25657]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:54.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:54 smithi017 ceph-mon[64584]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:54.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:54 smithi017 ceph-mon[67142]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:56.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:56 smithi129 ceph-mon[25657]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:56.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:56 smithi017 ceph-mon[64584]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:56.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:42:56] "GET /metrics HTTP/1.1" 200 38193 "" "Prometheus/2.33.4" 2024-03-26T04:42:56.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:56 smithi017 ceph-mon[67142]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:42:57.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:42:57 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:42:57.074+0000 7f879e59b700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-26T04:42:58.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:57 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:58.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:57 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:58.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:57 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:58.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:57 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:58.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:57 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:58.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:57 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.046 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:42:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:42:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:42:59.047 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:58 smithi129 ceph-mon[25657]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:59.047 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:58 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.047 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:58 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:58 smithi017 ceph-mon[64584]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:59.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:58 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:58 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:58 smithi017 ceph-mon[67142]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:58 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:58 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:42:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:42:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:59.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:42:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:00.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:42:59 smithi129 ceph-mon[25657]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:00 smithi017 ceph-mon[67142]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:00.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:00 smithi017 ceph-mon[64584]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:01.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:00 smithi129 ceph-mon[25657]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:43:02.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-26T04:43:02.344 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:02 smithi129 ceph-mon[25657]: Upgrade: It appears safe to stop mon.b 2024-03-26T04:43:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:02.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[67142]: Upgrade: It appears safe to stop mon.b 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:02.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:02.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:02.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:02.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T04:43:02.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-26T04:43:02.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:02 smithi017 ceph-mon[64584]: Upgrade: It appears safe to stop mon.b 2024-03-26T04:43:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:03 smithi129 ceph-mon[25657]: Upgrade: Updating mon.b 2024-03-26T04:43:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:03 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:03 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:03 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:03 smithi129 ceph-mon[25657]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:03.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:03 smithi129 ceph-mon[25657]: Deploying daemon mon.b on smithi129 2024-03-26T04:43:03.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[67142]: Upgrade: Updating mon.b 2024-03-26T04:43:03.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:03.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[67142]: Deploying daemon mon.b on smithi129 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[64584]: Upgrade: Updating mon.b 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:03.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:03 smithi017 ceph-mon[64584]: Deploying daemon mon.b on smithi129 2024-03-26T04:43:04.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:04 smithi129 systemd[1]: Stopping Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:43:04.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:04 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b[25652]: 2024-03-26T04:43:04.554+0000 7f4cefe1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:43:04.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:04 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b[25652]: 2024-03-26T04:43:04.554+0000 7f4cefe1f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:43:05.056 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:04 smithi129 podman[57729]: 2024-03-26 04:43:04.800176407 +0000 UTC m=+0.346657075 container died 685e2a3e6efc0a0111f57508765e00489fd8a18d9fa5e0efb3de7c384dcb6509 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-type=git, io.buildah.version=1.19.8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.expose-services=, ceph=True) 2024-03-26T04:43:05.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 podman[57729]: 2024-03-26 04:43:05.130293681 +0000 UTC m=+0.676774339 container cleanup 685e2a3e6efc0a0111f57508765e00489fd8a18d9fa5e0efb3de7c384dcb6509 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, ceph=True, version=8, RELEASE=HEAD, io.openshift.expose-services=, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc.) 2024-03-26T04:43:05.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 bash[57729]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b 2024-03-26T04:43:05.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 podman[57744]: 2024-03-26 04:43:05.298226594 +0000 UTC m=+0.497932318 container remove 685e2a3e6efc0a0111f57508765e00489fd8a18d9fa5e0efb3de7c384dcb6509 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , release=754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, name=centos-stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T04:43:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T04:43:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 systemd[1]: Stopped Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:43:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b.service: Consumed 9.458s CPU time. 2024-03-26T04:43:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 systemd[1]: Starting Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:43:06.252 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 podman[57874]: 2024-03-26 04:43:05.864508101 +0000 UTC m=+0.018902300 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:43:06.252 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:05 smithi129 podman[57874]: 2024-03-26 04:43:05.990899772 +0000 UTC m=+0.145293943 container create fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 podman[57874]: 2024-03-26 04:43:06.252295371 +0000 UTC m=+0.406689554 container init fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 podman[57874]: 2024-03-26 04:43:06.255486026 +0000 UTC m=+0.409880209 container start fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: pidfile_write: ignore empty --pid-file 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: load: jerasure load: lrc 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Git sha 0 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:43:06.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: DB SUMMARY 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: DB Session ID: GQ2WK4KD04QTJZ8HV394 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1540484 ; 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:43:06.647 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.env: 0x558e80a5bae0 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.info_log: 0x558e8152d920 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.statistics: (nil) 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.use_fsync: 0 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:43:06.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.db_log_dir: 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.wal_dir: 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.write_buffer_manager: 0x558e81500820 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:43:06.649 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.unordered_write: 0 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.row_cache: None 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.wal_filter: None 2024-03-26T04:43:06.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:43:06.651 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.wal_compression: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:43:06.652 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:43:06.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_open_files: -1 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Compression algorithms supported: 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kZSTD supported: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:43:06.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.merge_operator: 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_filter: None 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558e8152da20) 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:43:06.655 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_type: 0 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_index_type: 0 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_shortening: 1 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: checksum: 4 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: no_block_cache: 0 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache: 0x558e81531090 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_options: 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: capacity : 536870912 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_shard_bits : 4 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: strict_capacity_limit : 0 2024-03-26T04:43:06.656 INFO:journalctl@ceph.mon.b.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_compressed: (nil) 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: persistent_cache: (nil) 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size: 4096 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size_deviation: 10 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_restart_interval: 16 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_block_restart_interval: 1 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: metadata_block_size: 4096 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: partition_filters: 0 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: use_delta_encoding: 1 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: filter_policy: bloomfilter 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: whole_key_filtering: 1 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: verify_compression: 0 2024-03-26T04:43:06.657 INFO:journalctl@ceph.mon.b.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: format_version: 5 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_index_compression: 1 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_align: 0 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_auto_readahead_size: 262144 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: prepopulate_block_cache: 0 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression: NoCompression 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:43:06.658 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.num_levels: 7 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:43:06.659 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:43:06.660 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:43:06.661 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:43:06.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.ttl: 2592000 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:43:06.663 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-26T04:43:06.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14787, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fe59a370-bf68-42a9-a8f2-3d6607f58694 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428186285003, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428186289314, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 922221, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14792, "largest_seqno": 15289, "table_properties": {"data_size": 919202, "index_size": 1429, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 5141, "raw_average_key_size": 23, "raw_value_size": 914371, "raw_average_value_size": 4233, "num_data_blocks": 65, "num_entries": 216, "num_filter_entries": 216, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711428186, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fe59a370-bf68-42a9-a8f2-3d6607f58694", "db_session_id": "GQ2WK4KD04QTJZ8HV394", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711428186289407, "job": 1, "event": "recovery_finished"} 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558e81622000 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: DB pointer 0x558e8160c000 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:43:06.665 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** DB Stats ** 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: L0 1/0 900.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 414.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:43:06.666 INFO:journalctl@ceph.mon.b.smithi129.stdout: L6 1/0 11.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: Sum 2/0 12.22 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 414.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 414.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 414.3 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:43:06.667 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative compaction: 0.00 GB write, 135.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval compaction: 0.00 GB write, 135.75 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache BinnedLRUCache@0x558e81531090#2 capacity: 512.00 MB usage: 2.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: starting mon.b rank 2 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???) e3 preinit fsid 67a15696-eb29-11ee-95d0-87774f69a715 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).mds e1 new map 2024-03-26T04:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).mds e1 print_map 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: e1 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: legacy client fscid: -1 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: No filesystems configured 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 ceph-mon[57894]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 bash[57874]: fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 2024-03-26T04:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:06 smithi129 systemd[1]: Started Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:43:06.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:43:06] "GET /metrics HTTP/1.1" 200 38193 "" "Prometheus/2.33.4" 2024-03-26T04:43:09.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:43:08 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:43:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:43:12.193 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:43:12.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:11 smithi017 ceph-mon[67142]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:12.194 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:11 smithi017 ceph-mon[64584]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:12.865 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:43:12.865 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 16s ago 7m 16.9M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:43:12.865 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (100s) 31s ago 6m 43.2M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:43:12.865 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (71s) 16s ago 6m 49.8M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (61s) 31s ago 11m 447M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (95s) 16s ago 13m 491M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (46s) 16s ago 13m 42.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 starting - - - 2048M 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (21s) 16s ago 11m 17.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (2m) 16s ago 7m 15.2M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (114s) 31s ago 7m 18.5M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (11m) 16s ago 11m 53.8M 2737M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (10m) 16s ago 10m 55.8M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (10m) 16s ago 10m 52.4M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (10m) 16s ago 10m 53.7M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (9m) 31s ago 9m 55.9M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (9m) 31s ago 9m 51.7M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:43:12.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (8m) 31s ago 8m 54.6M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:43:12.867 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (8m) 31s ago 8m 54.1M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:43:12.867 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (66s) 31s ago 7m 53.7M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:43:12.867 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (6m) 16s ago 6m 72.7M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:43:12.867 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (6m) 31s ago 6m 76.3M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:43:12.867 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (6m) 16s ago 6m 73.3M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:43:12.867 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (6m) 31s ago 6m 73.6M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: mon.c calling monitor election 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: mon.a calling monitor election 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:13.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: fsmap 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: mgrmap e40: y(active, since 35s), standbys: x 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:13.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:12 smithi129 ceph-mon[57894]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: mon.c calling monitor election 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: mon.a calling monitor election 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: fsmap 2024-03-26T04:43:13.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: mgrmap e40: y(active, since 35s), standbys: x 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[67142]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: mon.c calling monitor election 2024-03-26T04:43:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: mon.a calling monitor election 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: fsmap 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: mgrmap e40: y(active, since 35s), standbys: x 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: from='client.44101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:13.208 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:12 smithi017 ceph-mon[64584]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:16.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:43:16.297+0000 7f879e59b700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-26T04:43:16.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:43:16] "GET /metrics HTTP/1.1" 200 38192 "" "Prometheus/2.33.4" 2024-03-26T04:43:19.006 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:43:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:43:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:43:19.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mon.c calling monitor election 2024-03-26T04:43:19.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mon.b calling monitor election 2024-03-26T04:43:19.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:19.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mon.c calling monitor election 2024-03-26T04:43:19.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:19.306 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:43:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-26T04:43:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 5 2024-03-26T04:43:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:43:19.308 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mon.c calling monitor election 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mon.b calling monitor election 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mon.c calling monitor election 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: fsmap 2024-03-26T04:43:19.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mgrmap e40: y(active, since 41s), standbys: x 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mon.a calling monitor election 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: fsmap 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: mgrmap e40: y(active, since 41s), standbys: x 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:43:19.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: overall HEALTH_OK 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mon.c calling monitor election 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mon.b calling monitor election 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mon.c calling monitor election 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: fsmap 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mgrmap e40: y(active, since 41s), standbys: x 2024-03-26T04:43:19.310 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mon.a calling monitor election 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: fsmap 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: mgrmap e40: y(active, since 41s), standbys: x 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: overall HEALTH_OK 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:19.311 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: fsmap 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mgrmap e40: y(active, since 41s), standbys: x 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mon.b (rank 2) addr [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] is down (out of quorum) 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mon.a calling monitor election 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:43:19.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: fsmap 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: mgrmap e40: y(active, since 41s), standbys: x 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: overall HEALTH_OK 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:19.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "5/23 daemons upgraded", 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:43:19.646 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:43:20.048 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:43:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:20 smithi129 ceph-mon[57894]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:20 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:20 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:20 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/3481859263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:20.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:20 smithi129 ceph-mon[57894]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:20.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:20 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1450232332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:43:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[64584]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/3481859263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[64584]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1450232332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[67142]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/3481859263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[67142]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:20 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1450232332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:43:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:22 smithi129 ceph-mon[57894]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:22 smithi017 ceph-mon[64584]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:22 smithi017 ceph-mon[67142]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: Reconfiguring mon.a (monmap changed)... 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:23 smithi129 ceph-mon[57894]: Reconfiguring daemon mon.a on smithi017 2024-03-26T04:43:23.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:23.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: Reconfiguring mon.a (monmap changed)... 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[64584]: Reconfiguring daemon mon.a on smithi017 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: Reconfiguring mon.a (monmap changed)... 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:23.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:23.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:23 smithi017 ceph-mon[67142]: Reconfiguring daemon mon.a on smithi017 2024-03-26T04:43:24.770 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[67142]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:24.770 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:24.770 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[64584]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:43:24.771 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:24 smithi129 ceph-mon[57894]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:43:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:43:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:25.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:25 smithi017 ceph-mon[64584]: Reconfiguring mgr.y (monmap changed)... 2024-03-26T04:43:25.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:25 smithi017 ceph-mon[64584]: Reconfiguring daemon mgr.y on smithi017 2024-03-26T04:43:25.679 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:25 smithi017 ceph-mon[67142]: Reconfiguring mgr.y (monmap changed)... 2024-03-26T04:43:25.679 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:25 smithi017 ceph-mon[67142]: Reconfiguring daemon mgr.y on smithi017 2024-03-26T04:43:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:25 smithi129 ceph-mon[57894]: Reconfiguring mgr.y (monmap changed)... 2024-03-26T04:43:25.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:25 smithi129 ceph-mon[57894]: Reconfiguring daemon mgr.y on smithi017 2024-03-26T04:43:26.646 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:43:26] "GET /metrics HTTP/1.1" 200 38263 "" "Prometheus/2.33.4" 2024-03-26T04:43:26.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:26.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:26.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:26.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:43:26.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[64584]: Reconfiguring daemon mon.c on smithi017 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:26.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:26 smithi017 ceph-mon[67142]: Reconfiguring daemon mon.c on smithi017 2024-03-26T04:43:27.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:27.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:26 smithi129 ceph-mon[57894]: Reconfiguring daemon mon.c on smithi017 2024-03-26T04:43:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.0 on smithi017 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:43:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:28.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:28 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.0 on smithi017 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.0 on smithi017 2024-03-26T04:43:29.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:43:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:43:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:43:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:43:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:43:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:30.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:29 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:30.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:30.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:30.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:43:30.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:31.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.1 on smithi017 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:31.143 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T04:43:31.144 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.1 on smithi017 2024-03-26T04:43:31.144 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:31.144 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:31.144 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:43:31.144 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:30 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.1 on smithi017 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:43:31.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:30 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:32.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:31 smithi129 ceph-mon[57894]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T04:43:32.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:31 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.2 on smithi017 2024-03-26T04:43:32.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:31 smithi017 ceph-mon[67142]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T04:43:32.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:31 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.2 on smithi017 2024-03-26T04:43:32.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:31 smithi017 ceph-mon[64584]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T04:43:32.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:31 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.2 on smithi017 2024-03-26T04:43:32.873 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:32.873 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:32.873 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:32.873 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T04:43:32.873 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.3 on smithi017 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:32.874 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:32 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.3 on smithi017 2024-03-26T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:33.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T04:43:33.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:43:33.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:33.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:32 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.3 on smithi017 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: Reconfiguring rgw.foo.smithi017.yqadkl (monmap changed)... 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:34 smithi129 ceph-mon[57894]: Reconfiguring daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:43:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: Reconfiguring rgw.foo.smithi017.yqadkl (monmap changed)... 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[64584]: Reconfiguring daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: Reconfiguring rgw.foo.smithi017.yqadkl (monmap changed)... 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:34.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:34 smithi017 ceph-mon[67142]: Reconfiguring daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:43:36.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:36.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: Reconfiguring rgw.smpl.smithi017.jmwydb (monmap changed)... 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[64584]: Reconfiguring daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: Reconfiguring rgw.smpl.smithi017.jmwydb (monmap changed)... 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:36.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-mon[67142]: Reconfiguring daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: Reconfiguring rgw.smpl.smithi017.jmwydb (monmap changed)... 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:36 smithi129 ceph-mon[57894]: Reconfiguring daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:43:36.674 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:43:36] "GET /metrics HTTP/1.1" 200 38263 "" "Prometheus/2.33.4" 2024-03-26T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:38.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: Reconfiguring daemon mon.b on smithi129 2024-03-26T04:43:38.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:38.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:38.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: Reconfiguring daemon mon.b on smithi129 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: Reconfiguring daemon mon.b on smithi129 2024-03-26T04:43:38.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:39.001 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:43:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:43:39.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-mon[57894]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:39.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:39.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:39.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:43:39.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:43:39.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[64584]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[67142]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:43:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:39.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:39 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:39.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:39 smithi129 ceph-mon[57894]: Reconfiguring mgr.x (monmap changed)... 2024-03-26T04:43:39.986 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:39 smithi129 ceph-mon[57894]: Reconfiguring daemon mgr.x on smithi129 2024-03-26T04:43:40.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:39 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:40.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:39 smithi017 ceph-mon[64584]: Reconfiguring mgr.x (monmap changed)... 2024-03-26T04:43:40.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:39 smithi017 ceph-mon[64584]: Reconfiguring daemon mgr.x on smithi129 2024-03-26T04:43:40.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:39 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:40.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:39 smithi017 ceph-mon[67142]: Reconfiguring mgr.x (monmap changed)... 2024-03-26T04:43:40.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:39 smithi017 ceph-mon[67142]: Reconfiguring daemon mgr.x on smithi129 2024-03-26T04:43:40.980 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:40 smithi129 ceph-mon[57894]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:40.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:40.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:40.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:43:40.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[67142]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[64584]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:41.107 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:43:41.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:41 smithi129 ceph-mon[57894]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T04:43:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:41 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.4 on smithi129 2024-03-26T04:43:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:41 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:41 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:42.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[64584]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.4 on smithi129 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[67142]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.4 on smithi129 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:42.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:41 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[64584]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[64584]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T04:43:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.5 on smithi129 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[67142]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[67142]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:42.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:42 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.5 on smithi129 2024-03-26T04:43:43.067 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:42 smithi129 ceph-mon[57894]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:43.068 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:42 smithi129 ceph-mon[57894]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T04:43:43.068 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:42 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:43:43.068 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:42 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:43.068 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:42 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.5 on smithi129 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:44.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:44 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.6 on smithi129 2024-03-26T04:43:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.6 on smithi129 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:44 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.6 on smithi129 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:46.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:46 smithi129 ceph-mon[57894]: Reconfiguring daemon osd.7 on smithi129 2024-03-26T04:43:46.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[67142]: Reconfiguring daemon osd.7 on smithi129 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:46.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T04:43:46.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:43:46.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:46.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-mon[64584]: Reconfiguring daemon osd.7 on smithi129 2024-03-26T04:43:46.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:43:46] "GET /metrics HTTP/1.1" 200 38200 "" "Prometheus/2.33.4" 2024-03-26T04:43:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:47 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:47 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:47 smithi129 ceph-mon[57894]: Reconfiguring rgw.foo.smithi129.pgirvy (monmap changed)... 2024-03-26T04:43:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:47 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:47 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:47.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:47 smithi129 ceph-mon[57894]: Reconfiguring daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:43:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[64584]: Reconfiguring rgw.foo.smithi129.pgirvy (monmap changed)... 2024-03-26T04:43:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:47.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[64584]: Reconfiguring daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[67142]: Reconfiguring rgw.foo.smithi129.pgirvy (monmap changed)... 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:47.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:47 smithi017 ceph-mon[67142]: Reconfiguring daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:43:48.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:48 smithi129 ceph-mon[57894]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:48.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:48.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:48.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:48.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:48.948 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:43:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:43:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[64584]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[67142]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:48.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:43:48.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:49.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:49 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:49.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:49 smithi017 ceph-mon[64584]: Reconfiguring rgw.smpl.smithi129.jvfyul (monmap changed)... 2024-03-26T04:43:49.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:49 smithi017 ceph-mon[64584]: Reconfiguring daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:43:49.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:49 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:49.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:49 smithi017 ceph-mon[67142]: Reconfiguring rgw.smpl.smithi129.jvfyul (monmap changed)... 2024-03-26T04:43:49.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:49 smithi017 ceph-mon[67142]: Reconfiguring daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:43:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:49 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:49 smithi129 ceph-mon[57894]: Reconfiguring rgw.smpl.smithi129.jvfyul (monmap changed)... 2024-03-26T04:43:50.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:49 smithi129 ceph-mon[57894]: Reconfiguring daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:43:50.418 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:43:50.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:50.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:50.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T04:43:50.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-26T04:43:50.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T04:43:50.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-26T04:43:50.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:50.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:50.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T04:43:51.099 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:43:51.099 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 54s ago 7m 16.9M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:43:51.099 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (2m) 34s ago 7m 45.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:43:51.099 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (109s) 54s ago 6m 49.8M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (99s) 34s ago 12m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (2m) 54s ago 13m 491M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (84s) 54s ago 13m 42.0M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (45s) 34s ago 12m 16.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (59s) 54s ago 12m 17.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 54s ago 8m 15.2M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (2m) 34s ago 8m 19.9M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (11m) 54s ago 11m 53.8M 2737M 17.2.0 e1d6a67b021e a7cca325c92f 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (11m) 54s ago 11m 55.8M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (11m) 54s ago 11m 52.4M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (10m) 54s ago 10m 53.7M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (10m) 34s ago 10m 56.4M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (9m) 34s ago 9m 52.2M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (9m) 34s ago 9m 55.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (9m) 34s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:43:51.100 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (104s) 34s ago 8m 56.0M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:43:51.101 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (7m) 54s ago 7m 72.7M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:43:51.101 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (6m) 34s ago 6m 76.6M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:43:51.101 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (6m) 54s ago 6m 73.3M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:43:51.101 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (6m) 34s ago 6m 73.9M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:43:51.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:51.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:51.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:51.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T04:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:51.510 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 5 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:43:51.511 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:43:51.858 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "5/23 daemons upgraded", 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:43:51.859 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:43:51.859 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all mon 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all crash 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: Upgrade: osd.0 is safe to restart 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1400832927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all mon 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all crash 2024-03-26T04:43:51.860 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: Upgrade: osd.0 is safe to restart 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:51.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:51 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1400832927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all mon 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all crash 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: Upgrade: osd.0 is safe to restart 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='client.34191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:52.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:43:52.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:43:52.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:52.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:51 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1400832927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:43:52.283 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:43:53.007 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[67142]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:53.007 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[67142]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:53.007 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[67142]: Upgrade: Updating osd.0 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[67142]: Deploying daemon osd.0 on smithi017 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/3354034918' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[64584]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[64584]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[64584]: Upgrade: Updating osd.0 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[64584]: Deploying daemon osd.0 on smithi017 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:53.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:52 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/3354034918' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:43:53.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:52 smithi129 ceph-mon[57894]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:53.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:52 smithi129 ceph-mon[57894]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:53.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:52 smithi129 ceph-mon[57894]: Upgrade: Updating osd.0 2024-03-26T04:43:53.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:52 smithi129 ceph-mon[57894]: Deploying daemon osd.0 on smithi017 2024-03-26T04:43:53.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:53.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:52 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/3354034918' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:43:54.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:53 smithi017 ceph-mon[64584]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:54.008 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:53 smithi017 ceph-mon[67142]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:53 smithi129 ceph-mon[57894]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:43:54.673 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:54 smithi017 systemd[1]: Stopping Ceph osd.0 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:43:54.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-mon[67142]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:54.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-mon[67142]: osd.0 marked itself down and dead 2024-03-26T04:43:54.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-mon[64584]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:54.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-mon[64584]: osd.0 marked itself down and dead 2024-03-26T04:43:54.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[32428]: 2024-03-26T04:43:54.672+0000 7f1f999dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:43:54.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[32428]: 2024-03-26T04:43:54.672+0000 7f1f999dc700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-26T04:43:54.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[32428]: 2024-03-26T04:43:54.672+0000 7f1f999dc700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:43:55.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:54 smithi129 ceph-mon[57894]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:55.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:54 smithi129 ceph-mon[57894]: osd.0 marked itself down and dead 2024-03-26T04:43:55.557 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:55 smithi017 podman[70053]: 2024-03-26 04:43:55.23478054 +0000 UTC m=+0.665036329 container died a7cca325c92fd076e3e84c933f3c6a7d02f8430f8773baf6251713978c054ba6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, name=centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD) 2024-03-26T04:43:55.866 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:43:55.866 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[67142]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.866 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[67142]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.866 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[67142]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T04:43:55.866 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[67142]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.866 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[67142]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:43:55.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[64584]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[64584]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[64584]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T04:43:55.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[64584]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:55 smithi017 ceph-mon[64584]: failed to encode map e107 with expected crc 2024-03-26T04:43:55.867 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:55 smithi017 podman[70053]: 2024-03-26 04:43:55.656370353 +0000 UTC m=+1.086626146 container cleanup a7cca325c92fd076e3e84c933f3c6a7d02f8430f8773baf6251713978c054ba6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, RELEASE=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, GIT_BRANCH=HEAD, release=754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T04:43:55.867 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:55 smithi017 bash[70053]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0 2024-03-26T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:55 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:55 smithi129 ceph-mon[57894]: failed to encode map e107 with expected crc 2024-03-26T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:55 smithi129 ceph-mon[57894]: failed to encode map e107 with expected crc 2024-03-26T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:55 smithi129 ceph-mon[57894]: osdmap e107: 8 total, 7 up, 8 in 2024-03-26T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:55 smithi129 ceph-mon[57894]: failed to encode map e107 with expected crc 2024-03-26T04:43:56.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:55 smithi129 ceph-mon[57894]: failed to encode map e107 with expected crc 2024-03-26T04:43:56.205 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:55 smithi017 podman[70080]: 2024-03-26 04:43:55.866467818 +0000 UTC m=+0.629215012 container remove a7cca325c92fd076e3e84c933f3c6a7d02f8430f8773baf6251713978c054ba6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, RELEASE=HEAD, version=8, architecture=x86_64, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container) 2024-03-26T04:43:56.561 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:43:56] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.33.4" 2024-03-26T04:43:56.561 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:56 smithi017 podman[70173]: 2024-03-26 04:43:56.267350947 +0000 UTC m=+0.111264580 container create 6ed062ec95d6ee5a00d5c564619c413972575e7b5c437589379b83909160bebe (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-26T04:43:56.561 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:56 smithi017 podman[70173]: 2024-03-26 04:43:56.173936769 +0000 UTC m=+0.017850426 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:43:56.561 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:56 smithi017 podman[70173]: 2024-03-26 04:43:56.495158919 +0000 UTC m=+0.339072560 container init 6ed062ec95d6ee5a00d5c564619c413972575e7b5c437589379b83909160bebe (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T04:43:56.561 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:56 smithi017 podman[70173]: 2024-03-26 04:43:56.498505269 +0000 UTC m=+0.342418905 container start 6ed062ec95d6ee5a00d5c564619c413972575e7b5c437589379b83909160bebe (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:43:56.561 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:56 smithi017 podman[70173]: 2024-03-26 04:43:56.561438448 +0000 UTC m=+0.405352088 container attach 6ed062ec95d6ee5a00d5c564619c413972575e7b5c437589379b83909160bebe (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T04:43:56.955 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:56 smithi017 podman[70173]: 2024-03-26 04:43:56.795657185 +0000 UTC m=+0.639570819 container died 6ed062ec95d6ee5a00d5c564619c413972575e7b5c437589379b83909160bebe (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T04:43:56.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: pgmap v43: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:43:56.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: osdmap e108: 8 total, 7 up, 8 in 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[67142]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: pgmap v43: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: osdmap e108: 8 total, 7 up, 8 in 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:56.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:56 smithi017 ceph-mon[64584]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: pgmap v43: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: osdmap e108: 8 total, 7 up, 8 in 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:56 smithi129 ceph-mon[57894]: failed to encode map e108 with expected crc 2024-03-26T04:43:57.705 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 podman[70199]: 2024-03-26 04:43:57.390627942 +0000 UTC m=+0.584885743 container remove 6ed062ec95d6ee5a00d5c564619c413972575e7b5c437589379b83909160bebe (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124) 2024-03-26T04:43:57.705 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-26T04:43:57.706 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 systemd[1]: Stopped Ceph osd.0 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:43:57.706 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.0.service: Consumed 7.469s CPU time. 2024-03-26T04:43:57.706 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 systemd[1]: Starting Ceph osd.0 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:43:58.127 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 podman[70302]: 2024-03-26 04:43:57.833564039 +0000 UTC m=+0.109624414 container create f749503229a157edafcf9bdb6d4644b58e741e9d288cc9dc45dac36a49100718 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:43:58.127 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:57 smithi017 podman[70302]: 2024-03-26 04:43:57.741834366 +0000 UTC m=+0.017894752 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:43:58.127 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 podman[70302]: 2024-03-26 04:43:58.052649081 +0000 UTC m=+0.328709457 container init f749503229a157edafcf9bdb6d4644b58e741e9d288cc9dc45dac36a49100718 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:43:58.127 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 podman[70302]: 2024-03-26 04:43:58.055985088 +0000 UTC m=+0.332045473 container start f749503229a157edafcf9bdb6d4644b58e741e9d288cc9dc45dac36a49100718 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:43:58.127 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 podman[70302]: 2024-03-26 04:43:58.127649448 +0000 UTC m=+0.403709837 container attach f749503229a157edafcf9bdb6d4644b58e741e9d288cc9dc45dac36a49100718 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T04:43:58.835 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-mon[64584]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:43:58.836 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-mon[67142]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:43:58.836 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T04:43:58.836 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T04:43:58.836 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T04:43:58.836 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T04:43:58.836 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 bash[70302]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate[70316]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T04:43:58.837 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:58 smithi017 podman[70302]: 2024-03-26 04:43:58.529906071 +0000 UTC m=+0.805966450 container died f749503229a157edafcf9bdb6d4644b58e741e9d288cc9dc45dac36a49100718 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5) 2024-03-26T04:43:59.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:58 smithi129 ceph-mon[57894]: pgmap v45: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:43:59.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:43:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:43:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:43:59.205 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 podman[70454]: 2024-03-26 04:43:59.05645014 +0000 UTC m=+0.518609676 container remove f749503229a157edafcf9bdb6d4644b58e741e9d288cc9dc45dac36a49100718 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T04:43:59.701 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 podman[70520]: 2024-03-26 04:43:59.175174354 +0000 UTC m=+0.018535738 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:43:59.701 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 podman[70520]: 2024-03-26 04:43:59.282658778 +0000 UTC m=+0.126020162 container create f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T04:43:59.701 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 podman[70520]: 2024-03-26 04:43:59.468105569 +0000 UTC m=+0.311466954 container init f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True) 2024-03-26T04:43:59.701 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 podman[70520]: 2024-03-26 04:43:59.471081305 +0000 UTC m=+0.314442689 container start f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-5e70124) 2024-03-26T04:43:59.701 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 ceph-osd[70540]: -- 172.21.15.17:0/4227265871 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556766c6b1e0 con 0x556766c3d000 2024-03-26T04:43:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:43:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:59.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:43:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:59.955 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 bash[70520]: f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca 2024-03-26T04:43:59.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:43:59 smithi017 systemd[1]: Started Ceph osd.0 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:44:00.046 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:43:59 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:00.455 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:44:00.088+0000 7f74b5b747c0 -1 Falling back to public interface 2024-03-26T04:44:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[67142]: pgmap v46: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 79/726 objects degraded (10.882%), 24 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:00.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[64584]: pgmap v46: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:00.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 79/726 objects degraded (10.882%), 24 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:00.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:00.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:01.061 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:00 smithi129 ceph-mon[57894]: pgmap v46: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:01.062 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:00 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 79/726 objects degraded (10.882%), 24 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:01.062 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:01.062 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:02.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:02 smithi017 ceph-mon[64584]: pgmap v47: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:02.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:02 smithi017 ceph-mon[67142]: pgmap v47: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:02.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:44:02 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:44:02.831+0000 7f74b5b747c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-26T04:44:02.956 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:44:02 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:44:02.857+0000 7f74b5b747c0 -1 osd.0 106 log_to_monitors true 2024-03-26T04:44:03.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:02 smithi129 ceph-mon[57894]: pgmap v47: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:04.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:03 smithi129 ceph-mon[57894]: from='osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:44:04.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:03 smithi129 ceph-mon[57894]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:44:04.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:03 smithi017 ceph-mon[64584]: from='osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:44:04.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:03 smithi017 ceph-mon[64584]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:44:04.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:03 smithi017 ceph-mon[67142]: from='osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:44:04.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:03 smithi017 ceph-mon[67142]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:44:04.206 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:44:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:44:03.777+0000 7f74a6fcc700 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:44:05.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: pgmap v48: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:05.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: osdmap e109: 8 total, 7 up, 8 in 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: from='osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[64584]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: pgmap v48: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:05.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: osdmap e109: 8 total, 7 up, 8 in 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: from='osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.075 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:04 smithi017 ceph-mon[67142]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: pgmap v48: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: osdmap e109: 8 total, 7 up, 8 in 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: from='osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:05.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:04 smithi129 ceph-mon[57894]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: failed to encode map e110 with expected crc 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: failed to encode map e110 with expected crc 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587] boot 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: failed to encode map e109 with expected crc 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: failed to encode map e110 with expected crc 2024-03-26T04:44:05.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: failed to encode map e110 with expected crc 2024-03-26T04:44:05.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587] boot 2024-03-26T04:44:05.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:44:05.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:44:05.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:05.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:05 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:06.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: failed to encode map e109 with expected crc 2024-03-26T04:44:06.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:44:06.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: failed to encode map e110 with expected crc 2024-03-26T04:44:06.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: failed to encode map e110 with expected crc 2024-03-26T04:44:06.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: osd.0 [v2:172.21.15.17:6802/368906587,v1:172.21.15.17:6803/368906587] boot 2024-03-26T04:44:06.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:44:06.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:44:06.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:06.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:05 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:06.701 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:44:06] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.33.4" 2024-03-26T04:44:06.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: pgmap v51: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:06.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[67142]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: pgmap v51: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:06.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:06.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:06 smithi017 ceph-mon[64584]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: pgmap v51: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 79/726 objects degraded (10.882%) 2024-03-26T04:44:07.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:07.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:06 smithi129 ceph-mon[57894]: failed to encode map e111 with expected crc 2024-03-26T04:44:08.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:07 smithi017 ceph-mon[64584]: pgmap v53: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:44:08.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:08.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:08.110 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:07 smithi017 ceph-mon[67142]: pgmap v53: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:44:08.110 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:08.110 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:07 smithi129 ceph-mon[57894]: pgmap v53: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:44:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:08.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:44:08.673+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:44:09.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:08 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:44:09.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:09.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.897 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:09.898 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:09 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:44:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:09.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:09.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:09 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:44:10.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:10 smithi129 ceph-mon[57894]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:10.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:10 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 79/726 objects degraded (10.882%), 24 pgs degraded) 2024-03-26T04:44:10.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:10 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:44:10.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:10 smithi017 ceph-mon[67142]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:10.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:10 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 79/726 objects degraded (10.882%), 24 pgs degraded) 2024-03-26T04:44:10.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:10 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:44:10.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:10 smithi017 ceph-mon[64584]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:10.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:10 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 79/726 objects degraded (10.882%), 24 pgs degraded) 2024-03-26T04:44:10.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:10 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:44:12.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:12 smithi129 ceph-mon[57894]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-26T04:44:12.958 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:12 smithi017 ceph-mon[64584]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-26T04:44:12.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:12 smithi017 ceph-mon[67142]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-03-26T04:44:14.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:14 smithi129 ceph-mon[57894]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:14.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:14 smithi017 ceph-mon[64584]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:14.952 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:14 smithi017 ceph-mon[67142]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:16 smithi017 ceph-mon[64584]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:44:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:16 smithi017 ceph-mon[67142]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:44:16.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:44:16] "GET /metrics HTTP/1.1" 200 38202 "" "Prometheus/2.33.4" 2024-03-26T04:44:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:16 smithi129 ceph-mon[57894]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:44:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:17 smithi017 ceph-mon[64584]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-26T04:44:18.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:17 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:18.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:17 smithi017 ceph-mon[67142]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-26T04:44:18.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:17 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:17 smithi129 ceph-mon[57894]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-26T04:44:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:17 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:19.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:18 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:19.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:44:19.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:18 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:19.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:18 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:20.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:19 smithi017 ceph-mon[64584]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:20.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:19 smithi017 ceph-mon[67142]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:20.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:19 smithi129 ceph-mon[57894]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:22.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:21 smithi017 ceph-mon[64584]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:22.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:21 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:22.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:21 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:22.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:21 smithi017 ceph-mon[67142]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:22.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:21 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:22.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:21 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:21 smithi129 ceph-mon[57894]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:21 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:22.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:21 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:22.650 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:44:23.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:22 smithi017 ceph-mon[64584]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:23.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:22 smithi017 ceph-mon[67142]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:23.319 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 19s ago 8m 24.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (2m) 66s ago 8m 45.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (2m) 19s ago 7m 52.2M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (2m) 66s ago 12m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (2m) 19s ago 14m 501M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (117s) 19s ago 14m 47.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (77s) 66s ago 12m 16.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (91s) 19s ago 13m 36.1M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 19s ago 9m 15.5M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (3m) 66s ago 8m 19.9M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (24s) 19s ago 12m 14.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (12m) 19s ago 12m 57.0M 2737M 17.2.0 e1d6a67b021e 535e3fec9673 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (11m) 19s ago 11m 53.7M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:44:23.320 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (11m) 19s ago 11m 54.9M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (10m) 66s ago 10m 56.4M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (10m) 66s ago 10m 52.2M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (10m) 66s ago 10m 55.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (9m) 66s ago 9m 54.5M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (2m) 66s ago 9m 56.0M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (7m) 19s ago 7m 73.0M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (7m) 66s ago 7m 76.6M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (7m) 19s ago 7m 73.6M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:44:23.321 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (7m) 66s ago 7m 73.9M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:44:23.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:22 smithi129 ceph-mon[57894]: from='client.44179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:23.729 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:44:23.729 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:44:23.729 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:44:23.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:23.729 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 1 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 6 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:44:23.730 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:44:24.079 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "6/23 daemons upgraded", 2024-03-26T04:44:24.080 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:44:24.080 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:44:24.080 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: Upgrade: osd.1 is safe to restart 2024-03-26T04:44:24.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/4037038452' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: Upgrade: osd.1 is safe to restart 2024-03-26T04:44:24.081 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:23 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/4037038452' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: from='client.34239 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T04:44:24.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: Upgrade: osd.1 is safe to restart 2024-03-26T04:44:24.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:23 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/4037038452' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:24.514 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:44:25.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:24 smithi017 ceph-mon[64584]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:25.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:24 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1415719913' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:44:25.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:24 smithi017 ceph-mon[67142]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:25.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:24 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1415719913' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:44:25.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:24 smithi129 ceph-mon[57894]: from='client.34251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:25.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:24 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1415719913' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:44:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:26 smithi129 ceph-mon[57894]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:26 smithi129 ceph-mon[57894]: Upgrade: Updating osd.1 2024-03-26T04:44:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:44:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:26 smithi129 ceph-mon[57894]: Deploying daemon osd.1 on smithi017 2024-03-26T04:44:26.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:44:26] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.33.4" 2024-03-26T04:44:26.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[64584]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[64584]: Upgrade: Updating osd.1 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[64584]: Deploying daemon osd.1 on smithi017 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[67142]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[67142]: Upgrade: Updating osd.1 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:26 smithi017 ceph-mon[67142]: Deploying daemon osd.1 on smithi017 2024-03-26T04:44:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:28 smithi017 ceph-mon[64584]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:28.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:28 smithi017 ceph-mon[67142]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:28.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:28 smithi129 ceph-mon[57894]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:29.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:44:29.171 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:28 smithi017 systemd[1]: Stopping Ceph osd.1 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:44:29.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:29.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-mon[64584]: osd.1 marked itself down and dead 2024-03-26T04:44:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:29.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-mon[67142]: osd.1 marked itself down and dead 2024-03-26T04:44:29.456 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[35408]: 2024-03-26T04:44:29.169+0000 7fa86d257700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:44:29.456 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[35408]: 2024-03-26T04:44:29.169+0000 7fa86d257700 -1 osd.1 111 *** Got signal Terminated *** 2024-03-26T04:44:29.456 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:29 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[35408]: 2024-03-26T04:44:29.169+0000 7fa86d257700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:44:29.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:29 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:29.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:29 smithi129 ceph-mon[57894]: osd.1 marked itself down and dead 2024-03-26T04:44:30.153 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:29 smithi017 podman[72541]: 2024-03-26 04:44:29.852905726 +0000 UTC m=+0.816286163 container stop 535e3fec9673f0786aa1c1cde05e9c6468a2060f49f814ab9b2ed59a5f355e17 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-03-26T04:44:30.154 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:29 smithi017 podman[72541]: 2024-03-26 04:44:29.878468318 +0000 UTC m=+0.841848748 container died 535e3fec9673f0786aa1c1cde05e9c6468a2060f49f814ab9b2ed59a5f355e17 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, maintainer=Guillaume Abrioux , release=754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container) 2024-03-26T04:44:30.429 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:30 smithi017 podman[72541]: 2024-03-26 04:44:30.269939742 +0000 UTC m=+1.233320171 container cleanup 535e3fec9673f0786aa1c1cde05e9c6468a2060f49f814ab9b2ed59a5f355e17 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, io.openshift.tags=base centos centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754) 2024-03-26T04:44:30.429 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:30 smithi017 bash[72541]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: osdmap e112: 8 total, 7 up, 8 in 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: osdmap e112: 8 total, 7 up, 8 in 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.432 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:30 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:30.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:44:30.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: osdmap e112: 8 total, 7 up, 8 in 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:30 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:30.705 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:30 smithi017 podman[72569]: 2024-03-26 04:44:30.429655773 +0000 UTC m=+0.570744314 container remove 535e3fec9673f0786aa1c1cde05e9c6468a2060f49f814ab9b2ed59a5f355e17 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, maintainer=Guillaume Abrioux , name=centos-stream, GIT_BRANCH=HEAD, ceph=True, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-26T04:44:31.116 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:30 smithi017 podman[72665]: 2024-03-26 04:44:30.822377852 +0000 UTC m=+0.108137638 container create 5adfc24b738df4371b6c2744983f461f70b18b8e7dec7ae4546ab0bcf2f8990c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T04:44:31.116 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:30 smithi017 podman[72665]: 2024-03-26 04:44:30.732243153 +0000 UTC m=+0.018002946 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:44:31.116 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 podman[72665]: 2024-03-26 04:44:31.033246719 +0000 UTC m=+0.319006497 container init 5adfc24b738df4371b6c2744983f461f70b18b8e7dec7ae4546ab0bcf2f8990c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:44:31.116 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 podman[72665]: 2024-03-26 04:44:31.036299017 +0000 UTC m=+0.322058804 container start 5adfc24b738df4371b6c2744983f461f70b18b8e7dec7ae4546ab0bcf2f8990c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:44:31.116 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 podman[72665]: 2024-03-26 04:44:31.116364044 +0000 UTC m=+0.402123834 container attach 5adfc24b738df4371b6c2744983f461f70b18b8e7dec7ae4546ab0bcf2f8990c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:44:31.370 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 podman[72665]: 2024-03-26 04:44:31.336730982 +0000 UTC m=+0.622490760 container died 5adfc24b738df4371b6c2744983f461f70b18b8e7dec7ae4546ab0bcf2f8990c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e112 with expected crc 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:31 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e112 with expected crc 2024-03-26T04:44:31.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T04:44:31.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e112 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:31.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:31 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:32.119 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 podman[72690]: 2024-03-26 04:44:31.895162607 +0000 UTC m=+0.553910321 container remove 5adfc24b738df4371b6c2744983f461f70b18b8e7dec7ae4546ab0bcf2f8990c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:44:32.119 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-26T04:44:32.120 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 systemd[1]: Stopped Ceph osd.1 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:44:32.120 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:31 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.1.service: Consumed 8.528s CPU time. 2024-03-26T04:44:32.120 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:32 smithi017 systemd[1]: Starting Ceph osd.1 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:44:32.371 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:32 smithi017 podman[72790]: 2024-03-26 04:44:32.236647517 +0000 UTC m=+0.017610695 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:44:32.371 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:32 smithi017 podman[72790]: 2024-03-26 04:44:32.354774551 +0000 UTC m=+0.135737767 container create 9dd3ff1d345164eadd432e18bfff405d42eaf1b54f15c7a59a32ab4384579e95 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:44:32.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:32 smithi129 ceph-mon[57894]: failed to encode map e113 with expected crc 2024-03-26T04:44:32.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:32 smithi129 ceph-mon[57894]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:44:32.665 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:32 smithi017 ceph-mon[67142]: failed to encode map e113 with expected crc 2024-03-26T04:44:32.665 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:32 smithi017 ceph-mon[67142]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:44:32.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:32 smithi017 ceph-mon[64584]: failed to encode map e113 with expected crc 2024-03-26T04:44:32.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:32 smithi017 ceph-mon[64584]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:44:32.666 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:32 smithi017 podman[72790]: 2024-03-26 04:44:32.582376322 +0000 UTC m=+0.363339499 container init 9dd3ff1d345164eadd432e18bfff405d42eaf1b54f15c7a59a32ab4384579e95 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T04:44:32.666 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:32 smithi017 podman[72790]: 2024-03-26 04:44:32.585595987 +0000 UTC m=+0.366559172 container start 9dd3ff1d345164eadd432e18bfff405d42eaf1b54f15c7a59a32ab4384579e95 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-5e70124) 2024-03-26T04:44:32.666 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:32 smithi017 podman[72790]: 2024-03-26 04:44:32.665539041 +0000 UTC m=+0.446502219 container attach 9dd3ff1d345164eadd432e18bfff405d42eaf1b54f15c7a59a32ab4384579e95 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:44:33.402 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T04:44:33.402 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T04:44:33.402 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T04:44:33.402 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 bash[72790]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate[72806]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T04:44:33.403 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 podman[72790]: 2024-03-26 04:44:33.058587342 +0000 UTC m=+0.839550528 container died 9dd3ff1d345164eadd432e18bfff405d42eaf1b54f15c7a59a32ab4384579e95 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T04:44:33.685 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 96/726 objects degraded (13.223%), 21 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:33.685 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:33 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 96/726 objects degraded (13.223%), 21 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:33 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 96/726 objects degraded (13.223%), 21 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:33.952 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 podman[72938]: 2024-03-26 04:44:33.686103889 +0000 UTC m=+0.621264052 container remove 9dd3ff1d345164eadd432e18bfff405d42eaf1b54f15c7a59a32ab4384579e95 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:44:33.952 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 podman[73002]: 2024-03-26 04:44:33.802938862 +0000 UTC m=+0.017968594 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:44:34.205 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:33 smithi017 podman[73002]: 2024-03-26 04:44:33.953578285 +0000 UTC m=+0.168608003 container create c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-26T04:44:34.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-mon[64584]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 1 stale+active+clean, 35 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 96/726 objects degraded (13.223%) 2024-03-26T04:44:34.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:34.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:34.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-mon[67142]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 1 stale+active+clean, 35 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 96/726 objects degraded (13.223%) 2024-03-26T04:44:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:34.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:34.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:34 smithi017 podman[73002]: 2024-03-26 04:44:34.239673476 +0000 UTC m=+0.454703195 container init c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T04:44:34.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:34 smithi017 podman[73002]: 2024-03-26 04:44:34.242762027 +0000 UTC m=+0.457791746 container start c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:44:34.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-osd[73021]: -- 172.21.15.17:0/541415618 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55764f1551e0 con 0x55764f126c00 2024-03-26T04:44:34.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:34 smithi017 bash[73002]: c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 2024-03-26T04:44:34.706 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:34 smithi017 systemd[1]: Started Ceph osd.1 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:44:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:34 smithi129 ceph-mon[57894]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 1 stale+active+clean, 35 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 96/726 objects degraded (13.223%) 2024-03-26T04:44:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:35.161 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:44:34.784+0000 7fc44859d7c0 -1 Falling back to public interface 2024-03-26T04:44:36.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:36 smithi017 ceph-mon[64584]: pgmap v69: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:36.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:36 smithi017 ceph-mon[67142]: pgmap v69: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:36.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:44:36] "GET /metrics HTTP/1.1" 200 38345 "" "Prometheus/2.33.4" 2024-03-26T04:44:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:36 smithi129 ceph-mon[57894]: pgmap v69: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:37.886 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:44:37.477+0000 7fc44859d7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-26T04:44:37.886 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:44:37.508+0000 7fc44859d7c0 -1 osd.1 111 log_to_monitors true 2024-03-26T04:44:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:37 smithi129 ceph-mon[57894]: pgmap v70: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:37 smithi129 ceph-mon[57894]: from='osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:44:38.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:37 smithi129 ceph-mon[57894]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:44:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[67142]: pgmap v70: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:38.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[67142]: from='osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:44:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[67142]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:44:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[64584]: pgmap v70: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:38.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[64584]: from='osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:44:38.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:37 smithi017 ceph-mon[64584]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:44:38.705 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:44:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:44:38.454+0000 7fc4399f5700 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:44:39.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:44:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:44:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: failed to encode map e114 with expected crc 2024-03-26T04:44:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T04:44:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: from='osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: failed to encode map e114 with expected crc 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: from='osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:39 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:44:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: failed to encode map e114 with expected crc 2024-03-26T04:44:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T04:44:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: from='osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:39.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:44:39.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:39.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:39 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:40.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: pgmap v72: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:40.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338] boot 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: pgmap v72: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338] boot 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:40.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:40.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: pgmap v72: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 103/726 objects degraded (14.187%) 2024-03-26T04:44:40.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: failed to encode map e114 with expected crc 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: osd.1 [v2:172.21.15.17:6810/3706805338,v1:172.21.15.17:6811/3706805338] boot 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:40.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:41.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:44:41.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[64584]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.657 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.658 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:41 smithi017 ceph-mon[67142]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:44:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:41.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:41 smithi129 ceph-mon[57894]: failed to encode map e116 with expected crc 2024-03-26T04:44:42.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:42 smithi017 ceph-mon[67142]: pgmap v75: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 103/726 objects degraded (14.187%) 2024-03-26T04:44:42.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:42 smithi017 ceph-mon[67142]: Health check update: Degraded data redundancy: 103/726 objects degraded (14.187%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:42 smithi017 ceph-mon[64584]: pgmap v75: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 103/726 objects degraded (14.187%) 2024-03-26T04:44:42.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:42 smithi017 ceph-mon[64584]: Health check update: Degraded data redundancy: 103/726 objects degraded (14.187%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:42 smithi129 ceph-mon[57894]: pgmap v75: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 103/726 objects degraded (14.187%) 2024-03-26T04:44:42.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:42 smithi129 ceph-mon[57894]: Health check update: Degraded data redundancy: 103/726 objects degraded (14.187%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T04:44:43.640 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:44:43.617+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: pgmap v76: 161 pgs: 9 active+undersized+degraded, 18 active+undersized, 134 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 45/726 objects degraded (6.198%) 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:44.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:44.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:44 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-26T04:44:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: pgmap v76: 161 pgs: 9 active+undersized+degraded, 18 active+undersized, 134 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 45/726 objects degraded (6.198%) 2024-03-26T04:44:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: pgmap v76: 161 pgs: 9 active+undersized+degraded, 18 active+undersized, 134 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 45/726 objects degraded (6.198%) 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:44.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:44 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-03-26T04:44:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:45 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 9 pgs degraded) 2024-03-26T04:44:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:45 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:44:45.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:45 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 9 pgs degraded) 2024-03-26T04:44:45.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:45 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:44:45.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:45 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 9 pgs degraded) 2024-03-26T04:44:45.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:45 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:44:46.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:46 smithi017 ceph-mon[64584]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-03-26T04:44:46.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:46 smithi017 ceph-mon[67142]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-03-26T04:44:46.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:44:46] "GET /metrics HTTP/1.1" 200 38412 "" "Prometheus/2.33.4" 2024-03-26T04:44:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:46 smithi129 ceph-mon[57894]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-03-26T04:44:48.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-26T04:44:48.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:44:48.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:48 smithi129 ceph-mon[57894]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:44:48.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:48 smithi017 ceph-mon[64584]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:44:48.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:48 smithi017 ceph-mon[67142]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:44:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:49 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:49.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:49 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:49.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:49 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:50 smithi129 ceph-mon[57894]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:44:50.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:50 smithi017 ceph-mon[64584]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:44:50.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:50 smithi017 ceph-mon[67142]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:44:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:52 smithi129 ceph-mon[57894]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-26T04:44:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[64584]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-26T04:44:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:52.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[67142]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-26T04:44:52.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:52.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:52.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:44:54.886 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:44:54.887 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:54 smithi017 ceph-mon[64584]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:54.887 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:54 smithi017 ceph-mon[67142]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:54.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:54 smithi129 ceph-mon[57894]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:55.553 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:44:55.553 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 16s ago 8m 24.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:44:55.553 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (3m) 99s ago 8m 45.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:44:55.553 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (2m) 16s ago 7m 52.4M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:44:55.553 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (2m) 99s ago 13m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (3m) 16s ago 14m 503M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (2m) 16s ago 14m 49.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (109s) 99s ago 13m 16.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 16s ago 13m 38.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 16s ago 9m 18.7M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (3m) 99s ago 9m 19.9M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (56s) 16s ago 12m 45.3M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (21s) 16s ago 12m 12.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (12m) 16s ago 12m 55.7M 2737M 17.2.0 e1d6a67b021e 1364a889a6a9 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (11m) 16s ago 11m 56.1M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:44:55.554 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (11m) 99s ago 11m 56.4M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (11m) 99s ago 11m 52.2M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (10m) 99s ago 10m 55.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (10m) 99s ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (2m) 99s ago 9m 56.0M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (8m) 16s ago 8m 73.2M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (8m) 99s ago 8m 76.6M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (7m) 16s ago 7m 73.7M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:44:55.555 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (7m) 99s ago 7m 73.9M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:44:55.959 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:44:55.959 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:44:55.960 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T04:44:55.961 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 7 2024-03-26T04:44:55.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:44:55.961 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:44:56.298 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:44:56.298 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "7/23 daemons upgraded", 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:44:56.299 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:44:56.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:44:56] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-03-26T04:44:56.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[64584]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:56.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[64584]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[64584]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[64584]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2277320356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:56.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[67142]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:56.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[67142]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[67142]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[67142]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:56 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2277320356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:56.708 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:44:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:56 smithi129 ceph-mon[57894]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:56 smithi129 ceph-mon[57894]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:56 smithi129 ceph-mon[57894]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:56 smithi129 ceph-mon[57894]: from='client.44233 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:56 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2277320356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:44:57.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:57 smithi129 ceph-mon[57894]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:57.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:57 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/631608689' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:44:57.920 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:57 smithi017 ceph-mon[67142]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:57.920 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:57 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/631608689' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:44:57.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:57 smithi017 ceph-mon[64584]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:44:57.920 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:57 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/631608689' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:44:58.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:58 smithi129 ceph-mon[57894]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:58.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:44:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:44:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:44:58.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:58 smithi017 ceph-mon[64584]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:58.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:58 smithi017 ceph-mon[67142]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:59 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:59 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:59 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:44:59 smithi129 ceph-mon[57894]: Upgrade: osd.2 is safe to restart 2024-03-26T04:44:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[64584]: Upgrade: osd.2 is safe to restart 2024-03-26T04:44:59.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:59.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:59.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T04:44:59.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:44:59 smithi017 ceph-mon[67142]: Upgrade: osd.2 is safe to restart 2024-03-26T04:45:00.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:00 smithi129 ceph-mon[57894]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:00.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:00.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:45:00.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:00.830 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[67142]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:00.830 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:00.830 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:45:00.830 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:00.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[64584]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:00.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:00.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:45:00.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:01.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:01 smithi017 ceph-mon[64584]: Upgrade: Updating osd.2 2024-03-26T04:45:01.652 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:01 smithi017 ceph-mon[64584]: Deploying daemon osd.2 on smithi017 2024-03-26T04:45:01.653 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:01 smithi017 ceph-mon[67142]: Upgrade: Updating osd.2 2024-03-26T04:45:01.653 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:01 smithi017 ceph-mon[67142]: Deploying daemon osd.2 on smithi017 2024-03-26T04:45:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:01 smithi129 ceph-mon[57894]: Upgrade: Updating osd.2 2024-03-26T04:45:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:01 smithi129 ceph-mon[57894]: Deploying daemon osd.2 on smithi017 2024-03-26T04:45:02.668 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:02 smithi017 ceph-mon[67142]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:02.668 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:02 smithi017 ceph-mon[64584]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:02.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:02 smithi129 ceph-mon[57894]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:03.583 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:03 smithi017 systemd[1]: Stopping Ceph osd.2 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:45:03.955 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[38319]: 2024-03-26T04:45:03.581+0000 7f7f2fc2c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:45:03.955 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[38319]: 2024-03-26T04:45:03.581+0000 7f7f2fc2c700 -1 osd.2 116 *** Got signal Terminated *** 2024-03-26T04:45:03.955 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[38319]: 2024-03-26T04:45:03.581+0000 7f7f2fc2c700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:45:04.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:04 smithi129 ceph-mon[57894]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:04.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:04 smithi129 ceph-mon[57894]: osd.2 marked itself down and dead 2024-03-26T04:45:04.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:04 smithi017 ceph-mon[64584]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:04.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:04 smithi017 ceph-mon[64584]: osd.2 marked itself down and dead 2024-03-26T04:45:04.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:04 smithi017 ceph-mon[67142]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:04.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:04 smithi017 ceph-mon[67142]: osd.2 marked itself down and dead 2024-03-26T04:45:05.358 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:04 smithi017 podman[75029]: 2024-03-26 04:45:04.991304289 +0000 UTC m=+1.503420748 container died 1364a889a6a9135f6e109e911a72b7e08e0d25a81129d38a767efbb045738125 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, vcs-type=git, release=754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T04:45:05.705 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:05 smithi017 podman[75029]: 2024-03-26 04:45:05.448224206 +0000 UTC m=+1.960340662 container cleanup 1364a889a6a9135f6e109e911a72b7e08e0d25a81129d38a767efbb045738125 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, distribution-scope=public, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, ceph=True) 2024-03-26T04:45:05.705 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:05 smithi017 bash[75029]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[64584]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:05 smithi017 ceph-mon[67142]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T04:45:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:45:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: osdmap e117: 8 total, 7 up, 8 in 2024-03-26T04:45:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: failed to encode map e117 with expected crc 2024-03-26T04:45:05.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:05 smithi129 ceph-mon[57894]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T04:45:06.067 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:05 smithi017 podman[75047]: 2024-03-26 04:45:05.744692044 +0000 UTC m=+0.751831695 container remove 1364a889a6a9135f6e109e911a72b7e08e0d25a81129d38a767efbb045738125 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, ceph=True, GIT_BRANCH=HEAD, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., name=centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=754, io.openshift.tags=base centos centos-stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8) 2024-03-26T04:45:06.328 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:06 smithi017 podman[75138]: 2024-03-26 04:45:06.067431289 +0000 UTC m=+0.111028308 container create a16a3a51a0c4e231103049b99fc71edfe0b668b1831122a9858b02385e5fb7b1 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:45:06.328 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:06 smithi017 podman[75138]: 2024-03-26 04:45:05.974139117 +0000 UTC m=+0.017736109 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:45:06.328 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:06 smithi017 podman[75138]: 2024-03-26 04:45:06.245028607 +0000 UTC m=+0.288625580 container init a16a3a51a0c4e231103049b99fc71edfe0b668b1831122a9858b02385e5fb7b1 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:45:06.328 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:06 smithi017 podman[75138]: 2024-03-26 04:45:06.248168854 +0000 UTC m=+0.291765830 container start a16a3a51a0c4e231103049b99fc71edfe0b668b1831122a9858b02385e5fb7b1 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:45:06.328 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:06 smithi017 podman[75138]: 2024-03-26 04:45:06.328459512 +0000 UTC m=+0.372056494 container attach a16a3a51a0c4e231103049b99fc71edfe0b668b1831122a9858b02385e5fb7b1 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T04:45:06.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:45:06] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-03-26T04:45:06.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:06 smithi017 podman[75138]: 2024-03-26 04:45:06.541870287 +0000 UTC m=+0.585467265 container died a16a3a51a0c4e231103049b99fc71edfe0b668b1831122a9858b02385e5fb7b1 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-26T04:45:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: pgmap v88: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:45:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[67142]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: pgmap v88: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:45:06.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.708 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:06 smithi017 ceph-mon[64584]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: pgmap v88: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:45:06.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:06.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:06 smithi129 ceph-mon[57894]: failed to encode map e118 with expected crc 2024-03-26T04:45:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 podman[75164]: 2024-03-26 04:45:07.157404257 +0000 UTC m=+0.610192291 container remove a16a3a51a0c4e231103049b99fc71edfe0b668b1831122a9858b02385e5fb7b1 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T04:45:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-26T04:45:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 systemd[1]: Stopped Ceph osd.2 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:45:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2.service: Consumed 6.579s CPU time. 2024-03-26T04:45:07.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 systemd[1]: Starting Ceph osd.2 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:45:07.888 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 podman[75270]: 2024-03-26 04:45:07.515677425 +0000 UTC m=+0.018114202 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:45:07.888 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 podman[75270]: 2024-03-26 04:45:07.625447718 +0000 UTC m=+0.127884479 container create 729490e1c38d51d9370172bde3c1ba4d764ab488ab4245069fae37cab28c266c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T04:45:07.888 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 podman[75270]: 2024-03-26 04:45:07.844656934 +0000 UTC m=+0.347093697 container init 729490e1c38d51d9370172bde3c1ba4d764ab488ab4245069fae37cab28c266c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T04:45:07.888 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 podman[75270]: 2024-03-26 04:45:07.847853272 +0000 UTC m=+0.350290027 container start 729490e1c38d51d9370172bde3c1ba4d764ab488ab4245069fae37cab28c266c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T04:45:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:07 smithi129 ceph-mon[57894]: pgmap v90: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:45:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:08.205 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:07 smithi017 podman[75270]: 2024-03-26 04:45:07.936407368 +0000 UTC m=+0.438844136 container attach 729490e1c38d51d9370172bde3c1ba4d764ab488ab4245069fae37cab28c266c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T04:45:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:07 smithi017 ceph-mon[64584]: pgmap v90: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:45:08.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:08.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:08.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:07 smithi017 ceph-mon[67142]: pgmap v90: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:45:08.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:08.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 bash[75270]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T04:45:08.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:45:08.707 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate[75284]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T04:45:08.707 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 podman[75270]: 2024-03-26 04:45:08.320015041 +0000 UTC m=+0.822451818 container died 729490e1c38d51d9370172bde3c1ba4d764ab488ab4245069fae37cab28c266c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:45:08.981 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:08.982 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:08 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:09.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:08 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:09.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:08 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:45:09.297 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:08 smithi017 podman[75406]: 2024-03-26 04:45:08.982280873 +0000 UTC m=+0.657145814 container remove 729490e1c38d51d9370172bde3c1ba4d764ab488ab4245069fae37cab28c266c (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:45:09.297 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 podman[75474]: 2024-03-26 04:45:09.102712321 +0000 UTC m=+0.018125670 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:45:09.297 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 podman[75474]: 2024-03-26 04:45:09.216452433 +0000 UTC m=+0.131865782 container create 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:45:09.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 podman[75474]: 2024-03-26 04:45:09.393857522 +0000 UTC m=+0.309270872 container init 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T04:45:09.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 podman[75474]: 2024-03-26 04:45:09.397303599 +0000 UTC m=+0.312716948 container start 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T04:45:09.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-osd[75493]: -- 172.21.15.17:0/457565500 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a797931e0 con 0x556a79765400 2024-03-26T04:45:09.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 bash[75474]: 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 2024-03-26T04:45:09.706 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 systemd[1]: Started Ceph osd.2 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:45:10.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:09 smithi129 ceph-mon[57894]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:10.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:09 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 66/726 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:10.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:10.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:10.205 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:45:09.935+0000 7efe97c717c0 -1 Falling back to public interface 2024-03-26T04:45:10.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[64584]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:10.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 66/726 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:10.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:10.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:10.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[67142]: pgmap v91: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:10.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 66/726 objects degraded (9.091%), 14 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:10.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:10.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:12.174 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:11 smithi017 ceph-mon[64584]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:12.175 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:11 smithi017 ceph-mon[67142]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:11 smithi129 ceph-mon[57894]: pgmap v92: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:12.901 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:12 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:45:12.628+0000 7efe97c717c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-26T04:45:12.902 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:12 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:45:12.653+0000 7efe97c717c0 -1 osd.2 116 log_to_monitors true 2024-03-26T04:45:13.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:12 smithi017 ceph-mon[64584]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:45:13.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:12 smithi017 ceph-mon[67142]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:45:13.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:12 smithi129 ceph-mon[57894]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:45:13.455 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:45:13.197+0000 7efe850c1700 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:45:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:45:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T04:45:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[67142]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T04:45:14.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:45:14.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:13 smithi017 ceph-mon[64584]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' 2024-03-26T04:45:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: pgmap v93: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:45:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T04:45:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:45:14.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: failed to encode map e119 with expected crc 2024-03-26T04:45:14.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:13 smithi129 ceph-mon[57894]: from='osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266]' entity='osd.2' 2024-03-26T04:45:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:45:15.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266] boot 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266] boot 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.210 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:14 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: osd.2 [v2:172.21.15.17:6818/493840266,v1:172.21.15.17:6819/493840266] boot 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:45:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:14 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: failed to encode map e120 with expected crc 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: pgmap v96: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:16 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: failed to encode map e120 with expected crc 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: pgmap v96: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: failed to encode map e120 with expected crc 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: pgmap v96: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.414 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:16.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:45:16] "GET /metrics HTTP/1.1" 200 38430 "" "Prometheus/2.33.4" 2024-03-26T04:45:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:17 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:17.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:17 smithi129 ceph-mon[57894]: failed to encode map e121 with expected crc 2024-03-26T04:45:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:17 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:17 smithi017 ceph-mon[64584]: failed to encode map e121 with expected crc 2024-03-26T04:45:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:17 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:17 smithi017 ceph-mon[67142]: failed to encode map e121 with expected crc 2024-03-26T04:45:18.140 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:18 smithi017 ceph-mon[64584]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:18.140 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:18 smithi017 ceph-mon[67142]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:18 smithi129 ceph-mon[57894]: pgmap v98: 161 pgs: 14 active+undersized+degraded, 32 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 66/726 objects degraded (9.091%) 2024-03-26T04:45:18.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:18 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:45:18.637+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-26T04:45:19.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:18.661Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-03-26T04:45:19.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:18.664Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:45:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:19.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:19.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:19 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-26T04:45:19.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:19.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-26T04:45:19.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:19.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:19 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-03-26T04:45:20.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:20 smithi129 ceph-mon[57894]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:20.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:20 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/726 objects degraded (9.091%), 14 pgs degraded) 2024-03-26T04:45:20.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:20 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:45:20.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:20 smithi017 ceph-mon[64584]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:20.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:20 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/726 objects degraded (9.091%), 14 pgs degraded) 2024-03-26T04:45:20.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:20 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:45:20.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:20 smithi017 ceph-mon[67142]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:20.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:20 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/726 objects degraded (9.091%), 14 pgs degraded) 2024-03-26T04:45:20.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:20 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:45:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:22 smithi129 ceph-mon[57894]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-26T04:45:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:22.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:22 smithi017 ceph-mon[64584]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-26T04:45:22.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:22.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:22.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:22 smithi017 ceph-mon[67142]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-26T04:45:22.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:22.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:24.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:24 smithi129 ceph-mon[57894]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:24 smithi017 ceph-mon[64584]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:24.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:24 smithi017 ceph-mon[67142]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:26.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:26 smithi017 ceph-mon[64584]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:45:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:26 smithi017 ceph-mon[67142]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:45:26.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:45:26] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.33.4" 2024-03-26T04:45:26.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:26 smithi129 ceph-mon[57894]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:45:27.078 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 13s ago 9m 24.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (3m) 2m ago 9m 45.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (3m) 13s ago 8m 52.6M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (3m) 2m ago 13m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (3m) 13s ago 15m 505M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:45:27.758 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (3m) 13s ago 15m 50.7M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (2m) 2m ago 14m 16.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 13s ago 14m 39.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 13s ago 10m 18.6M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (4m) 2m ago 10m 19.9M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (88s) 13s ago 13m 46.8M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (53s) 13s ago 13m 43.9M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (18s) 13s ago 12m 12.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (12m) 13s ago 12m 57.9M 2737M 17.2.0 e1d6a67b021e 4d7b7a8e449d 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (11m) 2m ago 11m 56.4M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (11m) 2m ago 11m 52.2M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (11m) 2m ago 11m 55.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (10m) 2m ago 10m 54.5M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (3m) 2m ago 10m 56.0M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (8m) 13s ago 8m 73.4M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (8m) 2m ago 8m 76.6M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:45:27.759 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (8m) 13s ago 8m 73.9M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:45:27.760 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (8m) 2m ago 8m 73.9M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:45:28.170 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:45:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:45:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:45:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:45:28.170 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:45:28.171 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[67142]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[67142]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[67142]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[67142]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[64584]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[64584]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.172 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[64584]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:27 smithi017 ceph-mon[64584]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:27 smithi129 ceph-mon[57894]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 0 op/s 2024-03-26T04:45:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:27 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:27 smithi129 ceph-mon[57894]: from='client.44263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:27 smithi129 ceph-mon[57894]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:27 smithi129 ceph-mon[57894]: from='client.44275 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "8/23 daemons upgraded", 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:45:28.515 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:45:28.920 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:45:28.938 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:45:29.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:29.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/256933144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:29.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[64584]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:29.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/751622121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:45:29.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:29.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/256933144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:29.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[67142]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:29.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:28 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/751622121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:45:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:28 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:28 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/256933144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:28 smithi129 ceph-mon[57894]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:45:29.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:28 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/751622121' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:45:30.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:29 smithi017 ceph-mon[64584]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:30.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:29 smithi017 ceph-mon[67142]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:30.296 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:29 smithi129 ceph-mon[57894]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:32.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:31 smithi017 ceph-mon[64584]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:32.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:31 smithi017 ceph-mon[67142]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:32.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:31 smithi129 ceph-mon[57894]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[64584]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:34.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[64584]: Upgrade: osd.3 is safe to restart 2024-03-26T04:45:34.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[67142]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:34.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:34.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:34.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:33 smithi017 ceph-mon[67142]: Upgrade: osd.3 is safe to restart 2024-03-26T04:45:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:33 smithi129 ceph-mon[57894]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:33 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:33 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T04:45:34.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:33 smithi129 ceph-mon[57894]: Upgrade: osd.3 is safe to restart 2024-03-26T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:36 smithi129 ceph-mon[57894]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:36 smithi129 ceph-mon[57894]: Upgrade: Updating osd.3 2024-03-26T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:36 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:36.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:36 smithi129 ceph-mon[57894]: Deploying daemon osd.3 on smithi017 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:45:36] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.33.4" 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[67142]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[67142]: Upgrade: Updating osd.3 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:36.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[67142]: Deploying daemon osd.3 on smithi017 2024-03-26T04:45:36.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[64584]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:36.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[64584]: Upgrade: Updating osd.3 2024-03-26T04:45:36.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:36.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:45:36.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:36.688 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:36 smithi017 ceph-mon[64584]: Deploying daemon osd.3 on smithi017 2024-03-26T04:45:37.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:37.668 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:37.668 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:38.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:38 smithi129 ceph-mon[57894]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:38.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:38 smithi017 ceph-mon[67142]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:38 smithi017 ceph-mon[64584]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:38.944 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:38 smithi017 systemd[1]: Stopping Ceph osd.3 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:45:39.145 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:45:39.205 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[41288]: 2024-03-26T04:45:38.990+0000 7f61a8968700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:45:39.205 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[41288]: 2024-03-26T04:45:38.990+0000 7f61a8968700 -1 osd.3 121 *** Got signal Terminated *** 2024-03-26T04:45:39.205 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[41288]: 2024-03-26T04:45:38.990+0000 7f61a8968700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:45:39.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:39 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:39.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:39 smithi129 ceph-mon[57894]: osd.3 marked itself down and dead 2024-03-26T04:45:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:39 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:39.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:39 smithi017 ceph-mon[64584]: osd.3 marked itself down and dead 2024-03-26T04:45:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:39 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:39.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:39 smithi017 ceph-mon[67142]: osd.3 marked itself down and dead 2024-03-26T04:45:40.191 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:39 smithi017 podman[77495]: 2024-03-26 04:45:39.937933183 +0000 UTC m=+1.077369942 container died 4d7b7a8e449d3348f94725167f20d68b1fc96886415f36fdf9fe4e46bdfd63ad (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, architecture=x86_64, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, release=754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, vcs-type=git, maintainer=Guillaume Abrioux , version=8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-26T04:45:40.455 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:40 smithi017 podman[77495]: 2024-03-26 04:45:40.299272443 +0000 UTC m=+1.438709202 container cleanup 4d7b7a8e449d3348f94725167f20d68b1fc96886415f36fdf9fe4e46bdfd63ad (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-03-26T04:45:40.455 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:40 smithi017 bash[77495]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: osdmap e122: 8 total, 7 up, 8 in 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[67142]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: osdmap e122: 8 total, 7 up, 8 in 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:40 smithi017 ceph-mon[64584]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: osdmap e122: 8 total, 7 up, 8 in 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:40 smithi129 ceph-mon[57894]: failed to encode map e122 with expected crc 2024-03-26T04:45:40.925 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:40 smithi017 podman[77521]: 2024-03-26 04:45:40.592723864 +0000 UTC m=+0.657853293 container remove 4d7b7a8e449d3348f94725167f20d68b1fc96886415f36fdf9fe4e46bdfd63ad (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T04:45:41.179 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:40 smithi017 podman[77616]: 2024-03-26 04:45:40.828384546 +0000 UTC m=+0.017538989 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:45:41.179 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:40 smithi017 podman[77616]: 2024-03-26 04:45:40.951898991 +0000 UTC m=+0.141053411 container create ecb8793c2a0f3167b9311f7260ad7c9c543bdead31219ce34aa27670f6b8df3f (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:45:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[64584]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:41 smithi017 ceph-mon[67142]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.457 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:41 smithi017 podman[77616]: 2024-03-26 04:45:41.179662942 +0000 UTC m=+0.368817368 container init ecb8793c2a0f3167b9311f7260ad7c9c543bdead31219ce34aa27670f6b8df3f (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124) 2024-03-26T04:45:41.457 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:41 smithi017 podman[77616]: 2024-03-26 04:45:41.183238024 +0000 UTC m=+0.372392451 container start ecb8793c2a0f3167b9311f7260ad7c9c543bdead31219ce34aa27670f6b8df3f (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T04:45:41.457 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:41 smithi017 podman[77616]: 2024-03-26 04:45:41.279438102 +0000 UTC m=+0.468592525 container attach ecb8793c2a0f3167b9311f7260ad7c9c543bdead31219ce34aa27670f6b8df3f (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T04:45:41.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T04:45:41.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:41 smithi129 ceph-mon[57894]: failed to encode map e123 with expected crc 2024-03-26T04:45:41.791 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:41 smithi017 podman[77616]: 2024-03-26 04:45:41.482559209 +0000 UTC m=+0.671713630 container died ecb8793c2a0f3167b9311f7260ad7c9c543bdead31219ce34aa27670f6b8df3f (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T04:45:42.395 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 podman[77643]: 2024-03-26 04:45:42.091652944 +0000 UTC m=+0.604212878 container remove ecb8793c2a0f3167b9311f7260ad7c9c543bdead31219ce34aa27670f6b8df3f (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T04:45:42.395 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T04:45:42.396 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 systemd[1]: Stopped Ceph osd.3 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:45:42.396 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3.service: Consumed 9.284s CPU time. 2024-03-26T04:45:42.396 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 systemd[1]: Starting Ceph osd.3 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:45:42.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:42 smithi017 ceph-mon[64584]: pgmap v112: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:45:42.906 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:42 smithi017 ceph-mon[67142]: pgmap v112: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:45:42.907 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 podman[77749]: 2024-03-26 04:45:42.450796835 +0000 UTC m=+0.017882642 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:45:42.907 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 podman[77749]: 2024-03-26 04:45:42.592929529 +0000 UTC m=+0.160015325 container create d144932a09bb12f12d4feda7189ab163cbf37c32b22a66fa37280af7b4f06ee2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:45:42.907 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:42 smithi129 ceph-mon[57894]: pgmap v112: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:45:43.206 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 podman[77749]: 2024-03-26 04:45:42.828794593 +0000 UTC m=+0.395880391 container init d144932a09bb12f12d4feda7189ab163cbf37c32b22a66fa37280af7b4f06ee2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T04:45:43.206 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 podman[77749]: 2024-03-26 04:45:42.831924437 +0000 UTC m=+0.399010243 container start d144932a09bb12f12d4feda7189ab163cbf37c32b22a66fa37280af7b4f06ee2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T04:45:43.206 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:42 smithi017 podman[77749]: 2024-03-26 04:45:42.937073788 +0000 UTC m=+0.504159601 container attach d144932a09bb12f12d4feda7189ab163cbf37c32b22a66fa37280af7b4f06ee2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T04:45:43.581 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 41/726 objects degraded (5.647%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:43.582 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 41/726 objects degraded (5.647%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:43.582 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T04:45:43.586 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T04:45:43.586 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T04:45:43.586 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T04:45:43.586 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T04:45:43.586 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T04:45:43.586 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 bash[77749]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate[77763]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T04:45:43.587 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 podman[77749]: 2024-03-26 04:45:43.301262118 +0000 UTC m=+0.868347918 container died d144932a09bb12f12d4feda7189ab163cbf37c32b22a66fa37280af7b4f06ee2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T04:45:43.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:43 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 41/726 objects degraded (5.647%), 12 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:44.125 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:43 smithi017 podman[77885]: 2024-03-26 04:45:43.87485116 +0000 UTC m=+0.566106050 container remove d144932a09bb12f12d4feda7189ab163cbf37c32b22a66fa37280af7b4f06ee2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux ) 2024-03-26T04:45:44.125 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 podman[77947]: 2024-03-26 04:45:43.990055402 +0000 UTC m=+0.017249906 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:45:44.125 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 podman[77947]: 2024-03-26 04:45:44.125729126 +0000 UTC m=+0.152923605 container create 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T04:45:44.394 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 podman[77947]: 2024-03-26 04:45:44.328275004 +0000 UTC m=+0.355469494 container init 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-26T04:45:44.394 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 podman[77947]: 2024-03-26 04:45:44.331579944 +0000 UTC m=+0.358774429 container start 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T04:45:44.395 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 ceph-osd[77966]: -- 172.21.15.17:0/3152312806 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d046e431e0 con 0x55d046e15000 2024-03-26T04:45:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:44 smithi017 ceph-mon[64584]: pgmap v113: 161 pgs: 12 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 41/726 objects degraded (5.647%) 2024-03-26T04:45:44.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:44 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:44.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:44 smithi017 ceph-mon[67142]: pgmap v113: 161 pgs: 12 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 41/726 objects degraded (5.647%) 2024-03-26T04:45:44.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:44 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:44.706 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 bash[77947]: 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 2024-03-26T04:45:44.706 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 systemd[1]: Started Ceph osd.3 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:45:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:44 smithi129 ceph-mon[57894]: pgmap v113: 161 pgs: 12 active+undersized+degraded, 13 stale+active+clean, 20 active+undersized, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 41/726 objects degraded (5.647%) 2024-03-26T04:45:44.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:44 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:45.205 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:45:44.875+0000 7f84b64ed7c0 -1 Falling back to public interface 2024-03-26T04:45:45.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:45 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:45.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:45 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:45.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:45 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:46.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:46 smithi017 ceph-mon[64584]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:46.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:46 smithi017 ceph-mon[67142]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:46.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:45:46] "GET /metrics HTTP/1.1" 200 38439 "" "Prometheus/2.33.4" 2024-03-26T04:45:46.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:46 smithi129 ceph-mon[57894]: pgmap v114: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:47.956 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:47 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:45:47.611+0000 7f84b64ed7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-26T04:45:47.956 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:47 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:45:47.644+0000 7f84b64ed7c0 -1 osd.3 121 log_to_monitors true 2024-03-26T04:45:48.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:48 smithi017 ceph-mon[64584]: pgmap v115: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:48.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:48 smithi017 ceph-mon[64584]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:45:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:48 smithi017 ceph-mon[67142]: pgmap v115: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:48 smithi017 ceph-mon[67142]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:45:48.706 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:45:48 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:45:48.485+0000 7f84a7945700 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:48 smithi129 ceph-mon[57894]: pgmap v115: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:48.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:48 smithi129 ceph-mon[57894]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:45:48.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:49.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:49 smithi129 ceph-mon[57894]: failed to encode map e124 with expected crc 2024-03-26T04:45:49.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:49.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:45:49.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T04:45:49.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:45:49.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:49.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:49.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[64584]: failed to encode map e124 with expected crc 2024-03-26T04:45:49.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:49.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:45:49.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T04:45:49.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: from='osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-03-26T04:45:49.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:49.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:49.958 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:49 smithi017 ceph-mon[67142]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: pgmap v117: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: failed to encode map e125 with expected crc 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: failed to encode map e125 with expected crc 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699] boot 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:45:50.758 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: pgmap v117: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: failed to encode map e125 with expected crc 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: failed to encode map e125 with expected crc 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699] boot 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:50.759 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:50 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: pgmap v117: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-03-26T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: failed to encode map e124 with expected crc 2024-03-26T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:45:50.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: failed to encode map e125 with expected crc 2024-03-26T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: failed to encode map e125 with expected crc 2024-03-26T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: osd.3 [v2:172.21.15.17:6826/3038126699,v1:172.21.15.17:6827/3038126699] boot 2024-03-26T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:50.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:50 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[64584]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.749 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.750 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.750 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:45:51.750 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.750 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.750 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.750 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:51 smithi017 ceph-mon[67142]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:51.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:51 smithi129 ceph-mon[57894]: failed to encode map e126 with expected crc 2024-03-26T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:52 smithi129 ceph-mon[57894]: pgmap v120: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/726 objects degraded (12.948%) 2024-03-26T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:52 smithi129 ceph-mon[57894]: Health check update: Degraded data redundancy: 94/726 objects degraded (12.948%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:52.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[64584]: pgmap v120: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/726 objects degraded (12.948%) 2024-03-26T04:45:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[64584]: Health check update: Degraded data redundancy: 94/726 objects degraded (12.948%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:52.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:52.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:52.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[67142]: pgmap v120: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/726 objects degraded (12.948%) 2024-03-26T04:45:52.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[67142]: Health check update: Degraded data redundancy: 94/726 objects degraded (12.948%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T04:45:52.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:52.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:53.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:45:53.514+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T04:45:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: pgmap v121: 161 pgs: 18 active+undersized+degraded, 30 active+undersized, 113 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73/726 objects degraded (10.055%) 2024-03-26T04:45:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:54.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:45:54.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:54 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T04:45:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: pgmap v121: 161 pgs: 18 active+undersized+degraded, 30 active+undersized, 113 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73/726 objects degraded (10.055%) 2024-03-26T04:45:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: pgmap v121: 161 pgs: 18 active+undersized+degraded, 30 active+undersized, 113 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 73/726 objects degraded (10.055%) 2024-03-26T04:45:54.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:45:54.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:54 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T04:45:55.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:55 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/726 objects degraded (10.055%), 18 pgs degraded) 2024-03-26T04:45:55.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:55 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:45:55.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:55 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/726 objects degraded (10.055%), 18 pgs degraded) 2024-03-26T04:45:55.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:55 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:45:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:55 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/726 objects degraded (10.055%), 18 pgs degraded) 2024-03-26T04:45:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:55 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:45:56.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:56 smithi017 ceph-mon[64584]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:56.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:56 smithi017 ceph-mon[67142]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:56.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:45:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:45:56] "GET /metrics HTTP/1.1" 200 38304 "" "Prometheus/2.33.4" 2024-03-26T04:45:56.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:56 smithi129 ceph-mon[57894]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:58.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:58 smithi017 ceph-mon[64584]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:58.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:58 smithi017 ceph-mon[67142]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:58.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:58 smithi129 ceph-mon[57894]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:45:58.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:45:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:45:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:45:59.293 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:45:59.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:45:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:45:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:45:59 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:59.959 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:45:59.959 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (5m) 11s ago 9m 23.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:45:59.959 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (4m) 2m ago 9m 45.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:45:59.959 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (3m) 11s ago 8m 53.0M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:45:59.959 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (3m) 2m ago 14m 449M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (4m) 11s ago 15m 507M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (3m) 11s ago 16m 49.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (2m) 2m ago 14m 16.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (3m) 11s ago 14m 38.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (5m) 11s ago 10m 15.8M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (4m) 2m ago 10m 19.9M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (2m) 11s ago 13m 47.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (86s) 11s ago 13m 44.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (50s) 11s ago 13m 50.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (15s) 11s ago 12m 12.5M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (12m) 2m ago 12m 56.4M 2992M 17.2.0 e1d6a67b021e 9c15e543aff3 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (12m) 2m ago 12m 52.2M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (11m) 2m ago 11m 55.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:45:59.960 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (11m) 2m ago 11m 54.5M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:45:59.961 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (3m) 2m ago 10m 56.0M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:45:59.961 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (9m) 11s ago 9m 73.5M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:45:59.961 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (9m) 2m ago 9m 76.6M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:45:59.961 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (8m) 11s ago 8m 74.0M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:45:59.961 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (8m) 2m ago 8m 73.9M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:46:00.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:00.366 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:46:00.366 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-26T04:46:00.366 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 9 2024-03-26T04:46:00.366 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:46:00.366 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:46:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[64584]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[64584]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[64584]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/929030817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:00.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[67142]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:00.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[67142]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:00.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[67142]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:00.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:00 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/929030817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:46:00.708 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "9/23 daemons upgraded", 2024-03-26T04:46:00.709 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:46:00.709 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:46:00.709 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:46:00.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:00 smithi129 ceph-mon[57894]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:00.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:00 smithi129 ceph-mon[57894]: from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:00.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:00 smithi129 ceph-mon[57894]: from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:00.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:00 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/929030817' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:01.112 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:46:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:01 smithi017 ceph-mon[64584]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:01 smithi017 ceph-mon[64584]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:01 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2660531020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:46:01.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:01 smithi017 ceph-mon[67142]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:01 smithi017 ceph-mon[67142]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:01 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2660531020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:46:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:01 smithi129 ceph-mon[57894]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:01 smithi129 ceph-mon[57894]: from='client.34377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:01.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:01 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2660531020' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:46:03.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:02 smithi129 ceph-mon[57894]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:03.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:02 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:03.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:02 smithi017 ceph-mon[64584]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:03.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:03.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:02 smithi017 ceph-mon[67142]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:03.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:05.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:04 smithi129 ceph-mon[57894]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:05.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:04 smithi017 ceph-mon[64584]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:05.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:04 smithi017 ceph-mon[67142]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:46:06.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:46:06] "GET /metrics HTTP/1.1" 200 38304 "" "Prometheus/2.33.4" 2024-03-26T04:46:07.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:06 smithi129 ceph-mon[57894]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-26T04:46:07.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:06 smithi017 ceph-mon[64584]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-26T04:46:07.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:06 smithi017 ceph-mon[67142]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-26T04:46:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:07 smithi129 ceph-mon[57894]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:07 smithi017 ceph-mon[64584]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:08.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:07 smithi017 ceph-mon[67142]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:08.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:08.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:09.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:08 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:09.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:08 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:46:09.114 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:08 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:46:09.115 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:08 smithi129 ceph-mon[57894]: Upgrade: osd.4 is safe to restart 2024-03-26T04:46:09.115 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:46:08 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:46:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi017:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.129:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T04:46:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:46:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:46:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[64584]: Upgrade: osd.4 is safe to restart 2024-03-26T04:46:09.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:09.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:46:09.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T04:46:09.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:08 smithi017 ceph-mon[67142]: Upgrade: osd.4 is safe to restart 2024-03-26T04:46:10.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:09 smithi017 ceph-mon[64584]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:10.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:09 smithi017 ceph-mon[67142]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:10.210 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:09 smithi129 ceph-mon[57894]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:11 smithi129 ceph-mon[57894]: Upgrade: Updating osd.4 2024-03-26T04:46:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:11 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:11 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:46:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:11 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:11.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:11 smithi129 ceph-mon[57894]: Deploying daemon osd.4 on smithi129 2024-03-26T04:46:11.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[64584]: Upgrade: Updating osd.4 2024-03-26T04:46:11.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:11.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:46:11.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:11.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[64584]: Deploying daemon osd.4 on smithi129 2024-03-26T04:46:11.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[67142]: Upgrade: Updating osd.4 2024-03-26T04:46:11.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:11.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:46:11.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:11.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:11 smithi017 ceph-mon[67142]: Deploying daemon osd.4 on smithi129 2024-03-26T04:46:12.529 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:12 smithi129 ceph-mon[57894]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:12.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:12 smithi017 ceph-mon[64584]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:12.607 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:12 smithi017 ceph-mon[67142]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:13.824 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:13 smithi129 systemd[1]: Stopping Ceph osd.4 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:46:14.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[29111]: 2024-03-26T04:46:13.823+0000 7f80f8327700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:46:14.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[29111]: 2024-03-26T04:46:13.823+0000 7f80f8327700 -1 osd.4 126 *** Got signal Terminated *** 2024-03-26T04:46:14.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:13 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[29111]: 2024-03-26T04:46:13.823+0000 7f80f8327700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:46:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:14 smithi017 ceph-mon[64584]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:14 smithi017 ceph-mon[64584]: osd.4 marked itself down and dead 2024-03-26T04:46:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:14 smithi017 ceph-mon[67142]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:14 smithi017 ceph-mon[67142]: osd.4 marked itself down and dead 2024-03-26T04:46:14.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:14 smithi129 ceph-mon[57894]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:14.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:14 smithi129 ceph-mon[57894]: osd.4 marked itself down and dead 2024-03-26T04:46:15.136 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:14 smithi129 podman[60704]: 2024-03-26 04:46:14.749590332 +0000 UTC m=+1.033912851 container died 9c15e543aff38cdbdb1ca62bd4b40b75aec35aa6ca86c255369f6f42ef18ec01 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, ceph=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-26T04:46:15.395 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:15 smithi129 podman[60704]: 2024-03-26 04:46:15.335286993 +0000 UTC m=+1.619609565 container cleanup 9c15e543aff38cdbdb1ca62bd4b40b75aec35aa6ca86c255369f6f42ef18ec01 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, distribution-scope=public, vcs-type=git, io.buildah.version=1.19.8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, ceph=True) 2024-03-26T04:46:15.395 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:15 smithi129 bash[60704]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4 2024-03-26T04:46:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:46:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: osdmap e127: 8 total, 7 up, 8 in 2024-03-26T04:46:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:15 smithi129 ceph-mon[57894]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: osdmap e127: 8 total, 7 up, 8 in 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[64584]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T04:46:15.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: osdmap e127: 8 total, 7 up, 8 in 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: failed to encode map e127 with expected crc 2024-03-26T04:46:15.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:15 smithi017 ceph-mon[67142]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T04:46:15.895 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:15 smithi129 podman[60732]: 2024-03-26 04:46:15.620083896 +0000 UTC m=+0.875003738 container remove 9c15e543aff38cdbdb1ca62bd4b40b75aec35aa6ca86c255369f6f42ef18ec01 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.openshift.expose-services=, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870, version=8) 2024-03-26T04:46:16.214 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:15 smithi129 podman[60829]: 2024-03-26 04:46:15.863586095 +0000 UTC m=+0.018261131 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:46:16.214 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:15 smithi129 podman[60829]: 2024-03-26 04:46:15.987992047 +0000 UTC m=+0.142667083 container create 25295b51cfdbf7b6169aeca6f1c6b089c1af840201845e6f0098fac9262484e7 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T04:46:16.512 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:16 smithi129 ceph-mon[57894]: pgmap v133: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:46:16.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:16 smithi129 ceph-mon[57894]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:16 smithi129 ceph-mon[57894]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:16 smithi129 ceph-mon[57894]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:16 smithi129 ceph-mon[57894]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.513 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:16 smithi129 podman[60829]: 2024-03-26 04:46:16.215371784 +0000 UTC m=+0.370046819 container init 25295b51cfdbf7b6169aeca6f1c6b089c1af840201845e6f0098fac9262484e7 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T04:46:16.513 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:16 smithi129 podman[60829]: 2024-03-26 04:46:16.218643077 +0000 UTC m=+0.373318126 container start 25295b51cfdbf7b6169aeca6f1c6b089c1af840201845e6f0098fac9262484e7 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124) 2024-03-26T04:46:16.513 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:16 smithi129 podman[60829]: 2024-03-26 04:46:16.281863433 +0000 UTC m=+0.436538490 container attach 25295b51cfdbf7b6169aeca6f1c6b089c1af840201845e6f0098fac9262484e7 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T04:46:16.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:46:16] "GET /metrics HTTP/1.1" 200 38444 "" "Prometheus/2.33.4" 2024-03-26T04:46:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[64584]: pgmap v133: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[64584]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[64584]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[64584]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[64584]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[67142]: pgmap v133: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[67142]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[67142]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[67142]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:16 smithi017 ceph-mon[67142]: failed to encode map e128 with expected crc 2024-03-26T04:46:16.895 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:16 smithi129 podman[60829]: 2024-03-26 04:46:16.513691964 +0000 UTC m=+0.668367010 container died 25295b51cfdbf7b6169aeca6f1c6b089c1af840201845e6f0098fac9262484e7 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T04:46:17.245 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 podman[60854]: 2024-03-26 04:46:17.211208662 +0000 UTC m=+0.691827075 container remove 25295b51cfdbf7b6169aeca6f1c6b089c1af840201845e6f0098fac9262484e7 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T04:46:17.245 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-26T04:46:17.245 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 systemd[1]: Stopped Ceph osd.4 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:46:17.245 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.4.service: Consumed 9.193s CPU time. 2024-03-26T04:46:17.645 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 systemd[1]: Starting Ceph osd.4 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:46:17.901 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 podman[60960]: 2024-03-26 04:46:17.554058279 +0000 UTC m=+0.018366461 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:46:17.901 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 podman[60960]: 2024-03-26 04:46:17.662522525 +0000 UTC m=+0.126830669 container create 31b2bff062c4b5f78122dca70577936f62f0d0a9b2f5fb4f90d5c5559bcc5d5e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:46:17.901 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 podman[60960]: 2024-03-26 04:46:17.898413269 +0000 UTC m=+0.362721435 container init 31b2bff062c4b5f78122dca70577936f62f0d0a9b2f5fb4f90d5c5559bcc5d5e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-26T04:46:18.281 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:17 smithi129 podman[60960]: 2024-03-26 04:46:17.901536212 +0000 UTC m=+0.365844378 container start 31b2bff062c4b5f78122dca70577936f62f0d0a9b2f5fb4f90d5c5559bcc5d5e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T04:46:18.282 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 podman[60960]: 2024-03-26 04:46:18.006918113 +0000 UTC m=+0.471226274 container attach 31b2bff062c4b5f78122dca70577936f62f0d0a9b2f5fb4f90d5c5559bcc5d5e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T04:46:18.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-mon[57894]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:46:18.645 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T04:46:18.645 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T04:46:18.645 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 bash[60960]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate[60975]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T04:46:18.646 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:18 smithi129 podman[60960]: 2024-03-26 04:46:18.364117812 +0000 UTC m=+0.828426007 container died 31b2bff062c4b5f78122dca70577936f62f0d0a9b2f5fb4f90d5c5559bcc5d5e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:46:18.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:18 smithi017 ceph-mon[64584]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:46:18.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:18 smithi017 ceph-mon[67142]: pgmap v135: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:46:19.288 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 podman[61094]: 2024-03-26 04:46:19.019455769 +0000 UTC m=+0.650654788 container remove 31b2bff062c4b5f78122dca70577936f62f0d0a9b2f5fb4f90d5c5559bcc5d5e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T04:46:19.288 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 podman[61159]: 2024-03-26 04:46:19.141870224 +0000 UTC m=+0.018067280 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:46:19.288 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 podman[61159]: 2024-03-26 04:46:19.245524918 +0000 UTC m=+0.121722005 container create cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:46:19.555 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:19 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:19.556 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 podman[61159]: 2024-03-26 04:46:19.456066114 +0000 UTC m=+0.332263163 container init cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:46:19.556 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 podman[61159]: 2024-03-26 04:46:19.459096069 +0000 UTC m=+0.335293114 container start cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T04:46:19.556 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 ceph-osd[61178]: -- 172.21.15.129:0/3663643849 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bb3f1451e0 con 0x55bb3f116c00 2024-03-26T04:46:19.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:19 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:19.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:19 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:19.895 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 bash[61159]: cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 2024-03-26T04:46:19.895 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:19 smithi129 systemd[1]: Started Ceph osd.4 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:46:20.288 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:20 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:46:20.005+0000 7f0507d357c0 -1 Falling back to public interface 2024-03-26T04:46:20.288 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:20 smithi129 ceph-mon[57894]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:20.288 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:20 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 125/726 objects degraded (17.218%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T04:46:20.623 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:20 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:20.623 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:20 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:20.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[64584]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 125/726 objects degraded (17.218%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[67142]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 125/726 objects degraded (17.218%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:20.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:20 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:22 smithi129 ceph-mon[57894]: pgmap v137: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:22.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:22 smithi017 ceph-mon[64584]: pgmap v137: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:22.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:22 smithi017 ceph-mon[67142]: pgmap v137: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:22.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:23.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:22 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:46:22.735+0000 7f0507d357c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-26T04:46:23.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:22 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:46:22.768+0000 7f0507d357c0 -1 osd.4 126 log_to_monitors true 2024-03-26T04:46:23.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:23 smithi129 ceph-mon[57894]: from='osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:46:23.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:23 smithi129 ceph-mon[57894]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:46:23.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:23 smithi017 ceph-mon[64584]: from='osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:46:23.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:23 smithi017 ceph-mon[64584]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:46:23.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:23 smithi017 ceph-mon[67142]: from='osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:46:23.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:23 smithi017 ceph-mon[67142]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:46:23.895 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:46:23 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:46:23.745+0000 7f04f5185700 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:46:24.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:24.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:46:24.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: failed to encode map e129 with expected crc 2024-03-26T04:46:24.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: failed to encode map e129 with expected crc 2024-03-26T04:46:24.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T04:46:24.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: from='osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:24.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:24.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: from='osd.4 ' entity='osd.4' 2024-03-26T04:46:24.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:24.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:24 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:24.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:24.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:46:24.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: failed to encode map e129 with expected crc 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: failed to encode map e129 with expected crc 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: from='osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: from='osd.4 ' entity='osd.4' 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: pgmap v138: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:46:24.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: failed to encode map e129 with expected crc 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: failed to encode map e129 with expected crc 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: from='osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: from='osd.4 ' entity='osd.4' 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:24.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:24 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:25.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:46:25.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: failed to encode map e130 with expected crc 2024-03-26T04:46:25.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295] boot 2024-03-26T04:46:25.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:46:25.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:46:25.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:25.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:25 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: failed to encode map e130 with expected crc 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295] boot 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:25.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: failed to encode map e130 with expected crc 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: osd.4 [v2:172.21.15.129:6800/2953742295,v1:172.21.15.129:6801/2953742295] boot 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:25.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:25 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: failed to encode map e130 with expected crc 2024-03-26T04:46:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: pgmap v141: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:46:26.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:26 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:46:26] "GET /metrics HTTP/1.1" 200 38449 "" "Prometheus/2.33.4" 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: failed to encode map e130 with expected crc 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: pgmap v141: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: failed to encode map e130 with expected crc 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: pgmap v141: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 125/726 objects degraded (17.218%) 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:26.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:26 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:27.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:27 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:27.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:27 smithi129 ceph-mon[57894]: failed to encode map e131 with expected crc 2024-03-26T04:46:27.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:27 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:27.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:27 smithi017 ceph-mon[64584]: failed to encode map e131 with expected crc 2024-03-26T04:46:27.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:27 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:27.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:27 smithi017 ceph-mon[67142]: failed to encode map e131 with expected crc 2024-03-26T04:46:28.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:28 smithi129 ceph-mon[57894]: pgmap v143: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 125/726 objects degraded (17.218%) 2024-03-26T04:46:28.656 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:28 smithi017 ceph-mon[64584]: pgmap v143: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 125/726 objects degraded (17.218%) 2024-03-26T04:46:28.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:28 smithi017 ceph-mon[67142]: pgmap v143: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 125/726 objects degraded (17.218%) 2024-03-26T04:46:28.955 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:28 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:46:28.656+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:29.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:29.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:29 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:46:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:29.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:29.957 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:29 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:46:30.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:30 smithi129 ceph-mon[57894]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:46:30.796 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:30 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 125/726 objects degraded (17.218%), 27 pgs degraded) 2024-03-26T04:46:30.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:30 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:46:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:30 smithi017 ceph-mon[64584]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:46:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:30 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 125/726 objects degraded (17.218%), 27 pgs degraded) 2024-03-26T04:46:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:30 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:46:30.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:30 smithi017 ceph-mon[67142]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:46:30.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:30 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 125/726 objects degraded (17.218%), 27 pgs degraded) 2024-03-26T04:46:30.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:30 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:46:31.483 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (5m) 43s ago 10m 23.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (5m) 8s ago 10m 54.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (4m) 43s ago 9m 53.0M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (4m) 8s ago 14m 450M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (4m) 43s ago 16m 507M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (4m) 43s ago 16m 49.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (3m) 8s ago 15m 35.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (3m) 43s ago 15m 38.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (5m) 43s ago 11m 15.8M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:46:32.155 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (5m) 8s ago 11m 19.1M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (2m) 43s ago 14m 47.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (118s) 43s ago 14m 44.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (83s) 43s ago 13m 50.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (48s) 43s ago 13m 12.5M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (13s) 8s ago 13m 12.7M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (12m) 8s ago 12m 57.7M 2992M 17.2.0 e1d6a67b021e 2c639cce911b 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (12m) 8s ago 12m 60.0M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (11m) 8s ago 11m 60.6M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (4m) 8s ago 11m 69.9M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (9m) 43s ago 9m 73.5M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (9m) 8s ago 9m 77.4M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (9m) 43s ago 9m 74.0M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:46:32.156 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (9m) 8s ago 9m 74.8M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 5 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 10 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:46:32.564 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:46:32.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:32 smithi129 ceph-mon[57894]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:46:32.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:32 smithi129 ceph-mon[57894]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:32.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:32 smithi129 ceph-mon[57894]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:46:32.904 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "10/23 daemons upgraded", 2024-03-26T04:46:32.905 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:46:32.905 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:46:32.905 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:46:32.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:32 smithi017 ceph-mon[64584]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:46:32.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:32 smithi017 ceph-mon[64584]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:32.905 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:32 smithi017 ceph-mon[64584]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:32.905 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:32 smithi017 ceph-mon[67142]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:46:32.905 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:32 smithi017 ceph-mon[67142]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:32.905 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:32 smithi017 ceph-mon[67142]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:33.312 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:46:33.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:33 smithi017 ceph-mon[64584]: from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:33.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:33 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/645589398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:33.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:33 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2691386490' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:46:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:33 smithi017 ceph-mon[67142]: from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:33 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/645589398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:33.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:33 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2691386490' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:46:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:33 smithi129 ceph-mon[57894]: from='client.44353 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:33 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/645589398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:46:33.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:33 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2691386490' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:46:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:34 smithi129 ceph-mon[57894]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-03-26T04:46:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:34 smithi129 ceph-mon[57894]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:34 smithi017 ceph-mon[64584]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-03-26T04:46:34.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:34 smithi017 ceph-mon[64584]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:34 smithi017 ceph-mon[67142]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-03-26T04:46:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:34 smithi017 ceph-mon[67142]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:46:36.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:36 smithi017 ceph-mon[64584]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:46:36.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:36 smithi017 ceph-mon[67142]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:46:36.706 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:46:36] "GET /metrics HTTP/1.1" 200 38449 "" "Prometheus/2.33.4" 2024-03-26T04:46:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:36 smithi129 ceph-mon[57894]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:46:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:37 smithi129 ceph-mon[57894]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:46:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:38.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:38.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[64584]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:46:38.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:38.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:38.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[67142]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:46:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:38.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:38 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:39.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:38 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:39.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:38 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:40.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:39 smithi017 ceph-mon[64584]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:40.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:39 smithi017 ceph-mon[67142]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:40.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:39 smithi129 ceph-mon[57894]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:42.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:41 smithi017 ceph-mon[64584]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:42.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:41 smithi017 ceph-mon[67142]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:42.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:41 smithi129 ceph-mon[57894]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:44.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[64584]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:44.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:44.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:44.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[64584]: Upgrade: osd.5 is safe to restart 2024-03-26T04:46:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[67142]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:44.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:43 smithi017 ceph-mon[67142]: Upgrade: osd.5 is safe to restart 2024-03-26T04:46:44.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:43 smithi129 ceph-mon[57894]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:44.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:43 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:44.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:43 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T04:46:44.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:43 smithi129 ceph-mon[57894]: Upgrade: osd.5 is safe to restart 2024-03-26T04:46:46.552 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:46 smithi129 ceph-mon[57894]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:46.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:46 smithi129 ceph-mon[57894]: Upgrade: Updating osd.5 2024-03-26T04:46:46.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:46.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:46:46.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:46.553 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:46 smithi129 ceph-mon[57894]: Deploying daemon osd.5 on smithi129 2024-03-26T04:46:46.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:46:46] "GET /metrics HTTP/1.1" 200 38447 "" "Prometheus/2.33.4" 2024-03-26T04:46:46.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[64584]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[64584]: Upgrade: Updating osd.5 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[64584]: Deploying daemon osd.5 on smithi129 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[67142]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[67142]: Upgrade: Updating osd.5 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:46.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:46:46.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:46.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:46 smithi017 ceph-mon[67142]: Deploying daemon osd.5 on smithi129 2024-03-26T04:46:48.529 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:48 smithi129 ceph-mon[57894]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:48.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:48 smithi017 ceph-mon[64584]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:48.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:48 smithi017 ceph-mon[67142]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:48.780 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:48 smithi129 systemd[1]: Stopping Ceph osd.5 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:46:49.145 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[31996]: 2024-03-26T04:46:48.814+0000 7fd2871fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:46:49.145 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[31996]: 2024-03-26T04:46:48.814+0000 7fd2871fe700 -1 osd.5 131 *** Got signal Terminated *** 2024-03-26T04:46:49.145 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[31996]: 2024-03-26T04:46:48.814+0000 7fd2871fe700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:46:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:49 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:49.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:49 smithi129 ceph-mon[57894]: osd.5 marked itself down and dead 2024-03-26T04:46:49.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:49 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:49.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:49 smithi017 ceph-mon[64584]: osd.5 marked itself down and dead 2024-03-26T04:46:49.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:49 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:49.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:49 smithi017 ceph-mon[67142]: osd.5 marked itself down and dead 2024-03-26T04:46:50.132 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:49 smithi129 podman[63049]: 2024-03-26 04:46:49.811378908 +0000 UTC m=+1.112513601 container died 2c639cce911b5f798075063b924efcaad602ab9192e5c9dd7897944db1afbebb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, io.openshift.tags=base centos centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , release=754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-type=git, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-03-26T04:46:50.395 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:50 smithi129 podman[63049]: 2024-03-26 04:46:50.206207279 +0000 UTC m=+1.507341953 container cleanup 2c639cce911b5f798075063b924efcaad602ab9192e5c9dd7897944db1afbebb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, vcs-type=git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, RELEASE=HEAD, vendor=Red Hat, Inc.) 2024-03-26T04:46:50.395 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:50 smithi129 bash[63049]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5 2024-03-26T04:46:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:46:50.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: osdmap e132: 8 total, 7 up, 8 in 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:50 smithi129 ceph-mon[57894]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:46:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: osdmap e132: 8 total, 7 up, 8 in 2024-03-26T04:46:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.707 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[64584]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: osdmap e132: 8 total, 7 up, 8 in 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:50 smithi017 ceph-mon[67142]: failed to encode map e132 with expected crc 2024-03-26T04:46:50.785 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:50 smithi129 podman[63075]: 2024-03-26 04:46:50.474765491 +0000 UTC m=+0.659645975 container remove 2c639cce911b5f798075063b924efcaad602ab9192e5c9dd7897944db1afbebb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.k8s.display-name=CentOS Stream 8) 2024-03-26T04:46:51.078 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:50 smithi129 podman[63168]: 2024-03-26 04:46:50.689151922 +0000 UTC m=+0.017886824 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:46:51.078 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:50 smithi129 podman[63168]: 2024-03-26 04:46:50.817369652 +0000 UTC m=+0.146104526 container create 66f41e702a7b87fd75c5aeac2c00fe99a543fd7bb052d6a77d6218d7791d1a04 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:46:51.078 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 podman[63168]: 2024-03-26 04:46:51.078632879 +0000 UTC m=+0.407367764 container init 66f41e702a7b87fd75c5aeac2c00fe99a543fd7bb052d6a77d6218d7791d1a04 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T04:46:51.394 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 podman[63168]: 2024-03-26 04:46:51.082534023 +0000 UTC m=+0.411268913 container start 66f41e702a7b87fd75c5aeac2c00fe99a543fd7bb052d6a77d6218d7791d1a04 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:46:51.395 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 podman[63168]: 2024-03-26 04:46:51.170279259 +0000 UTC m=+0.499014153 container attach 66f41e702a7b87fd75c5aeac2c00fe99a543fd7bb052d6a77d6218d7791d1a04 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: osdmap e133: 8 total, 7 up, 8 in 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:51 smithi129 ceph-mon[57894]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.645 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 podman[63168]: 2024-03-26 04:46:51.395539624 +0000 UTC m=+0.724274517 container died 66f41e702a7b87fd75c5aeac2c00fe99a543fd7bb052d6a77d6218d7791d1a04 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T04:46:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: osdmap e133: 8 total, 7 up, 8 in 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[64584]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: osdmap e133: 8 total, 7 up, 8 in 2024-03-26T04:46:51.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:51.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:51 smithi017 ceph-mon[67142]: failed to encode map e133 with expected crc 2024-03-26T04:46:52.120 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 podman[63194]: 2024-03-26 04:46:51.899011314 +0000 UTC m=+0.494936738 container remove 66f41e702a7b87fd75c5aeac2c00fe99a543fd7bb052d6a77d6218d7791d1a04 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T04:46:52.120 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-26T04:46:52.120 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 systemd[1]: Stopped Ceph osd.5 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:46:52.121 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:51 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.5.service: Consumed 8.458s CPU time. 2024-03-26T04:46:52.121 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:52 smithi129 systemd[1]: Starting Ceph osd.5 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:46:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:52 smithi129 ceph-mon[57894]: pgmap v157: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:52.395 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:52 smithi129 podman[63301]: 2024-03-26 04:46:52.240049145 +0000 UTC m=+0.018202209 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:46:52.395 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:52 smithi129 podman[63301]: 2024-03-26 04:46:52.341755383 +0000 UTC m=+0.119908441 container create 894b43bb4a764bea32d0b1147382f8f912a363f829422bd8e2dba1ed10ba5567 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:46:52.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:52 smithi017 ceph-mon[67142]: pgmap v157: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:52.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:52.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:52.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:52 smithi017 ceph-mon[64584]: pgmap v157: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:52.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:52.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:52.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:52 smithi129 podman[63301]: 2024-03-26 04:46:52.560970363 +0000 UTC m=+0.339123422 container init 894b43bb4a764bea32d0b1147382f8f912a363f829422bd8e2dba1ed10ba5567 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T04:46:52.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:52 smithi129 podman[63301]: 2024-03-26 04:46:52.564004862 +0000 UTC m=+0.342157930 container start 894b43bb4a764bea32d0b1147382f8f912a363f829422bd8e2dba1ed10ba5567 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T04:46:52.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:52 smithi129 podman[63301]: 2024-03-26 04:46:52.652496005 +0000 UTC m=+0.430649065 container attach 894b43bb4a764bea32d0b1147382f8f912a363f829422bd8e2dba1ed10ba5567 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:46:53.344 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T04:46:53.344 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T04:46:53.344 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 bash[63301]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate[63317]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T04:46:53.345 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 podman[63301]: 2024-03-26 04:46:53.024207592 +0000 UTC m=+0.802360651 container died 894b43bb4a764bea32d0b1147382f8f912a363f829422bd8e2dba1ed10ba5567 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:46:53.640 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:53 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 20/726 objects degraded (2.755%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T04:46:53.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:53 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 20/726 objects degraded (2.755%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T04:46:53.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:53 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 20/726 objects degraded (2.755%), 6 pgs degraded (PG_DEGRADED) 2024-03-26T04:46:53.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 podman[63440]: 2024-03-26 04:46:53.640688983 +0000 UTC m=+0.611739331 container remove 894b43bb4a764bea32d0b1147382f8f912a363f829422bd8e2dba1ed10ba5567 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T04:46:53.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 podman[63504]: 2024-03-26 04:46:53.757153801 +0000 UTC m=+0.017977660 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:46:53.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:53 smithi129 podman[63504]: 2024-03-26 04:46:53.858480033 +0000 UTC m=+0.119303892 container create 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:46:54.361 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:54 smithi129 podman[63504]: 2024-03-26 04:46:54.09438764 +0000 UTC m=+0.355211495 container init 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T04:46:54.361 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:54 smithi129 podman[63504]: 2024-03-26 04:46:54.09732714 +0000 UTC m=+0.358150997 container start 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T04:46:54.361 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:54 smithi129 ceph-osd[63523]: -- 172.21.15.129:0/1570752362 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5650369851e0 con 0x565036957400 2024-03-26T04:46:54.361 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:54 smithi129 bash[63504]: 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d 2024-03-26T04:46:54.361 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:54 smithi129 systemd[1]: Started Ceph osd.5 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:46:54.641 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:54 smithi129 ceph-mon[57894]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s; 20/726 objects degraded (2.755%) 2024-03-26T04:46:54.641 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:54.641 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:54 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:54 smithi017 ceph-mon[64584]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s; 20/726 objects degraded (2.755%) 2024-03-26T04:46:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:54.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:54 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:54.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:54 smithi017 ceph-mon[67142]: pgmap v158: 161 pgs: 6 active+undersized+degraded, 16 stale+active+clean, 9 active+undersized, 130 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s; 20/726 objects degraded (2.755%) 2024-03-26T04:46:54.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:54.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:54 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:54.895 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:46:54.640+0000 7fa53a64e7c0 -1 Falling back to public interface 2024-03-26T04:46:56.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:56 smithi129 ceph-mon[57894]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 118 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:46:56.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:56 smithi017 ceph-mon[64584]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 118 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:46:56.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:56 smithi017 ceph-mon[67142]: pgmap v159: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 0 B/s wr, 118 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:46:56.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:46:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:46:56] "GET /metrics HTTP/1.1" 200 38450 "" "Prometheus/2.33.4" 2024-03-26T04:46:57.645 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:46:57.356+0000 7fa53a64e7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-26T04:46:57.645 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:46:57.387+0000 7fa53a64e7c0 -1 osd.5 131 log_to_monitors true 2024-03-26T04:46:58.645 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:46:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:46:58.396+0000 7fa52baa6700 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:46:58.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:58 smithi129 ceph-mon[57894]: pgmap v160: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 118 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:46:58.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:58 smithi129 ceph-mon[57894]: from='osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:46:58.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:58 smithi129 ceph-mon[57894]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:46:58.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:58 smithi017 ceph-mon[64584]: pgmap v160: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 118 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:46:58.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:58 smithi017 ceph-mon[64584]: from='osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:46:58.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:58 smithi017 ceph-mon[64584]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:46:58.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:58 smithi017 ceph-mon[67142]: pgmap v160: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 118 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:46:58.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:58 smithi017 ceph-mon[67142]: from='osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:46:58.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:58 smithi017 ceph-mon[67142]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: from='osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: osdmap e134: 8 total, 7 up, 8 in 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:59.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:46:59 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: from='osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: osdmap e134: 8 total, 7 up, 8 in 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:59.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: from='osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: osdmap e134: 8 total, 7 up, 8 in 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:46:59.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:46:59 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: pgmap v162: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 163 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393] boot 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.413 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:00 smithi129 ceph-mon[57894]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:47:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: pgmap v162: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 163 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:47:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393] boot 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.706 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[64584]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: pgmap v162: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 163 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: osd.5 [v2:172.21.15.129:6808/501882393,v1:172.21.15.129:6809/501882393] boot 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.707 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e134 with expected crc 2024-03-26T04:47:00.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: failed to encode map e135 with expected crc 2024-03-26T04:47:00.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:00.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:00 smithi017 ceph-mon[67142]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:47:01.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:01 smithi129 ceph-mon[57894]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:01 smithi129 ceph-mon[57894]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:01 smithi129 ceph-mon[57894]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:01 smithi129 ceph-mon[57894]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[64584]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[64584]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[64584]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[64584]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[67142]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[67142]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[67142]: failed to encode map e136 with expected crc 2024-03-26T04:47:01.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:01 smithi017 ceph-mon[67142]: failed to encode map e136 with expected crc 2024-03-26T04:47:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:02 smithi129 ceph-mon[57894]: pgmap v165: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:47:02.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:02 smithi129 ceph-mon[57894]: Health check update: Degraded data redundancy: 79/726 objects degraded (10.882%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T04:47:02.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:02 smithi017 ceph-mon[64584]: pgmap v165: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:47:02.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:02 smithi017 ceph-mon[64584]: Health check update: Degraded data redundancy: 79/726 objects degraded (10.882%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T04:47:02.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:02 smithi017 ceph-mon[67142]: pgmap v165: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 79/726 objects degraded (10.882%) 2024-03-26T04:47:02.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:02 smithi017 ceph-mon[67142]: Health check update: Degraded data redundancy: 79/726 objects degraded (10.882%), 20 pgs degraded (PG_DEGRADED) 2024-03-26T04:47:03.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:47:03.192+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:47:03.717 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 75s ago 10m 23.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (5m) 5s ago 10m 54.4M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (5m) 75s ago 9m 53.0M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (4m) 5s ago 15m 450M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (5m) 75s ago 16m 507M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (4m) 75s ago 17m 49.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:47:04.385 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (3m) 5s ago 15m 36.9M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (4m) 75s ago 15m 38.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 75s ago 11m 15.8M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (5m) 5s ago 11m 19.3M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (3m) 75s ago 15m 47.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (2m) 75s ago 14m 44.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (115s) 75s ago 14m 50.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (80s) 75s ago 13m 12.5M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (45s) 5s ago 13m 53.2M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (10s) 5s ago 13m 12.6M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 0e80c6fa2668 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (12m) 5s ago 12m 63.4M 2992M 17.2.0 e1d6a67b021e bc70f63c923d 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (12m) 5s ago 12m 62.9M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (4m) 5s ago 11m 72.1M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:47:04.386 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (10m) 75s ago 10m 73.5M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:47:04.387 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (10m) 5s ago 10m 77.7M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:47:04.387 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (10m) 75s ago 10m 74.0M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:47:04.387 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (9m) 5s ago 9m 75.1M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: pgmap v166: 161 pgs: 16 active+undersized+degraded, 33 active+undersized, 112 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 66/726 objects degraded (9.091%); 0 B/s, 1 objects/s recovering 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[67142]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: pgmap v166: 161 pgs: 16 active+undersized+degraded, 33 active+undersized, 112 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 66/726 objects degraded (9.091%); 0 B/s, 1 objects/s recovering 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:04.388 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:47:04.389 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:04 smithi017 ceph-mon[64584]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: pgmap v166: 161 pgs: 16 active+undersized+degraded, 33 active+undersized, 112 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s; 66/726 objects degraded (9.091%); 0 B/s, 1 objects/s recovering 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:04.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:04.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:04.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-03-26T04:47:04.396 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:04 smithi129 ceph-mon[57894]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:04.795 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 6 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:47:04.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:04.797 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:47:04.797 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T04:47:04.797 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 11 2024-03-26T04:47:04.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:47:04.797 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:47:05.133 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "11/23 daemons upgraded", 2024-03-26T04:47:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:47:05.134 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:47:05.134 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:47:05.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:05 smithi017 ceph-mon[64584]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:05.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:05 smithi017 ceph-mon[64584]: from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:05.134 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:05 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1769406122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:05.134 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:05 smithi017 ceph-mon[67142]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:05.135 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:05 smithi017 ceph-mon[67142]: from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:05.135 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:05 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1769406122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:05.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:05 smithi129 ceph-mon[57894]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:05.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:05 smithi129 ceph-mon[57894]: from='client.44395 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:05.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:05 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1769406122' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:05.536 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:47:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:06 smithi129 ceph-mon[57894]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s; 0 B/s, 7 objects/s recovering 2024-03-26T04:47:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:06 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/726 objects degraded (9.091%), 16 pgs degraded) 2024-03-26T04:47:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:06 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:47:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:06 smithi129 ceph-mon[57894]: from='client.44407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:06.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:06 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/915773930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:47:06.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[67142]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s; 0 B/s, 7 objects/s recovering 2024-03-26T04:47:06.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/726 objects degraded (9.091%), 16 pgs degraded) 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[67142]: from='client.44407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/915773930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[64584]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s; 0 B/s, 7 objects/s recovering 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/726 objects degraded (9.091%), 16 pgs degraded) 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[64584]: from='client.44407 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:06.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/915773930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:47:06.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:47:06] "GET /metrics HTTP/1.1" 200 38450 "" "Prometheus/2.33.4" 2024-03-26T04:47:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:07 smithi129 ceph-mon[57894]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s; 0 B/s, 6 objects/s recovering 2024-03-26T04:47:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:08.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:08.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:07 smithi017 ceph-mon[67142]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s; 0 B/s, 6 objects/s recovering 2024-03-26T04:47:08.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:08.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:07 smithi017 ceph-mon[64584]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 33 op/s; 0 B/s, 6 objects/s recovering 2024-03-26T04:47:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:08.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:09.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:08 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:09.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:08 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:09.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:08 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:10.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:09 smithi017 ceph-mon[64584]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 151 op/s; 0 B/s, 5 objects/s recovering 2024-03-26T04:47:10.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:09 smithi017 ceph-mon[67142]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 151 op/s; 0 B/s, 5 objects/s recovering 2024-03-26T04:47:10.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:09 smithi129 ceph-mon[57894]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 151 op/s; 0 B/s, 5 objects/s recovering 2024-03-26T04:47:12.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:11 smithi017 ceph-mon[64584]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 0 B/s, 4 objects/s recovering 2024-03-26T04:47:12.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:11 smithi017 ceph-mon[67142]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 0 B/s, 4 objects/s recovering 2024-03-26T04:47:12.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:11 smithi129 ceph-mon[57894]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 0 B/s wr, 136 op/s; 0 B/s, 4 objects/s recovering 2024-03-26T04:47:13.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:13.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:13.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:14.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:13 smithi129 ceph-mon[57894]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 0 B/s, 4 objects/s recovering 2024-03-26T04:47:14.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:13 smithi017 ceph-mon[64584]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 0 B/s, 4 objects/s recovering 2024-03-26T04:47:14.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:13 smithi017 ceph-mon[67142]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 0 B/s, 4 objects/s recovering 2024-03-26T04:47:16.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:15 smithi129 ceph-mon[57894]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s; 0 B/s, 3 objects/s recovering 2024-03-26T04:47:16.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:15 smithi017 ceph-mon[64584]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s; 0 B/s, 3 objects/s recovering 2024-03-26T04:47:16.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:15 smithi017 ceph-mon[67142]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s; 0 B/s, 3 objects/s recovering 2024-03-26T04:47:16.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:47:16] "GET /metrics HTTP/1.1" 200 38459 "" "Prometheus/2.33.4" 2024-03-26T04:47:18.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:17 smithi129 ceph-mon[57894]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-03-26T04:47:18.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:17 smithi017 ceph-mon[64584]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-03-26T04:47:18.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:17 smithi017 ceph-mon[67142]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-03-26T04:47:19.032 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:18 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:19.032 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:18 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:19.032 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:18 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:19.032 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:18 smithi129 ceph-mon[57894]: Upgrade: osd.6 is safe to restart 2024-03-26T04:47:19.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:19.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:19.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:19.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[67142]: Upgrade: osd.6 is safe to restart 2024-03-26T04:47:19.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:19.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:19.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T04:47:19.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:18 smithi017 ceph-mon[64584]: Upgrade: osd.6 is safe to restart 2024-03-26T04:47:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:19 smithi129 ceph-mon[57894]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-03-26T04:47:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:47:20.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:19 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[64584]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-03-26T04:47:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:47:20.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:20.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[67142]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-03-26T04:47:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:47:20.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:19 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:21.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:20 smithi129 ceph-mon[57894]: Upgrade: Updating osd.6 2024-03-26T04:47:21.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:20 smithi129 ceph-mon[57894]: Deploying daemon osd.6 on smithi129 2024-03-26T04:47:21.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:20 smithi017 ceph-mon[67142]: Upgrade: Updating osd.6 2024-03-26T04:47:21.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:20 smithi017 ceph-mon[67142]: Deploying daemon osd.6 on smithi129 2024-03-26T04:47:21.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:20 smithi017 ceph-mon[64584]: Upgrade: Updating osd.6 2024-03-26T04:47:21.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:20 smithi017 ceph-mon[64584]: Deploying daemon osd.6 on smithi129 2024-03-26T04:47:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:21 smithi129 ceph-mon[57894]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:22.019 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:21 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:21 smithi017 ceph-mon[67142]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:22.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:21 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:21 smithi017 ceph-mon[64584]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:22.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:21 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:23.395 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:23 smithi129 systemd[1]: Stopping Ceph osd.6 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:47:23.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:23 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[34880]: 2024-03-26T04:47:23.494+0000 7fdf9816b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:47:23.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:23 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[34880]: 2024-03-26T04:47:23.494+0000 7fdf9816b700 -1 osd.6 136 *** Got signal Terminated *** 2024-03-26T04:47:23.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:23 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[34880]: 2024-03-26T04:47:23.494+0000 7fdf9816b700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:47:24.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:23 smithi129 ceph-mon[57894]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:24.343 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:23 smithi129 ceph-mon[57894]: osd.6 marked itself down and dead 2024-03-26T04:47:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:23 smithi017 ceph-mon[67142]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:23 smithi017 ceph-mon[67142]: osd.6 marked itself down and dead 2024-03-26T04:47:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:23 smithi017 ceph-mon[64584]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:23 smithi017 ceph-mon[64584]: osd.6 marked itself down and dead 2024-03-26T04:47:24.645 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:24 smithi129 podman[65394]: 2024-03-26 04:47:24.34358024 +0000 UTC m=+0.950691189 container died bc70f63c923d73b12b817007adb41876b839822a224883da0e95754542be55e0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, RELEASE=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.openshift.expose-services=, ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T04:47:24.995 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:24 smithi129 podman[65394]: 2024-03-26 04:47:24.728492072 +0000 UTC m=+1.335603019 container cleanup bc70f63c923d73b12b817007adb41876b839822a224883da0e95754542be55e0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, version=8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., distribution-scope=public, name=centos-stream, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T04:47:24.996 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:24 smithi129 bash[65394]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6 2024-03-26T04:47:25.337 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:24 smithi129 podman[65413]: 2024-03-26 04:47:24.996459247 +0000 UTC m=+0.649529054 container remove bc70f63c923d73b12b817007adb41876b839822a224883da0e95754542be55e0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, RELEASE=HEAD, release=754, vcs-type=git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, distribution-scope=public) 2024-03-26T04:47:25.337 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:24 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:47:25.337 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:24 smithi129 ceph-mon[57894]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.337 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:24 smithi129 ceph-mon[57894]: osdmap e137: 8 total, 7 up, 8 in 2024-03-26T04:47:25.337 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:24 smithi129 ceph-mon[57894]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.337 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:24 smithi129 ceph-mon[57894]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:47:25.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[64584]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[64584]: osdmap e137: 8 total, 7 up, 8 in 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[64584]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[64584]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[67142]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[67142]: osdmap e137: 8 total, 7 up, 8 in 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[67142]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:24 smithi017 ceph-mon[67142]: failed to encode map e137 with expected crc 2024-03-26T04:47:25.607 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:25 smithi129 podman[65503]: 2024-03-26 04:47:25.240406134 +0000 UTC m=+0.018211714 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:47:25.607 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:25 smithi129 podman[65503]: 2024-03-26 04:47:25.339103192 +0000 UTC m=+0.116908751 container create 95a137f6b0682bf2f10f76ab95b62a44e3b3fbec051f0793975af11a8d3fba06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T04:47:25.607 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:25 smithi129 podman[65503]: 2024-03-26 04:47:25.533127067 +0000 UTC m=+0.310932627 container init 95a137f6b0682bf2f10f76ab95b62a44e3b3fbec051f0793975af11a8d3fba06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T04:47:25.608 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:25 smithi129 podman[65503]: 2024-03-26 04:47:25.536324074 +0000 UTC m=+0.314129633 container start 95a137f6b0682bf2f10f76ab95b62a44e3b3fbec051f0793975af11a8d3fba06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:47:25.608 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:25 smithi129 podman[65503]: 2024-03-26 04:47:25.608125607 +0000 UTC m=+0.385931180 container attach 95a137f6b0682bf2f10f76ab95b62a44e3b3fbec051f0793975af11a8d3fba06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T04:47:25.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:25 smithi129 podman[65503]: 2024-03-26 04:47:25.826042795 +0000 UTC m=+0.603848342 container died 95a137f6b0682bf2f10f76ab95b62a44e3b3fbec051f0793975af11a8d3fba06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:47:26.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:26 smithi129 ceph-mon[57894]: pgmap v178: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:47:26.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:26 smithi129 ceph-mon[57894]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:26 smithi129 ceph-mon[57894]: osdmap e138: 8 total, 7 up, 8 in 2024-03-26T04:47:26.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:26 smithi129 ceph-mon[57894]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.336 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:26 smithi129 ceph-mon[57894]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[64584]: pgmap v178: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[64584]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[64584]: osdmap e138: 8 total, 7 up, 8 in 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[64584]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[64584]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[67142]: pgmap v178: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[67142]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[67142]: osdmap e138: 8 total, 7 up, 8 in 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[67142]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-mon[67142]: failed to encode map e138 with expected crc 2024-03-26T04:47:26.645 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 podman[65529]: 2024-03-26 04:47:26.337007494 +0000 UTC m=+0.502066499 container remove 95a137f6b0682bf2f10f76ab95b62a44e3b3fbec051f0793975af11a8d3fba06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:47:26.645 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T04:47:26.645 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 systemd[1]: Stopped Ceph osd.6 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:47:26.645 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.6.service: Consumed 8.371s CPU time. 2024-03-26T04:47:26.645 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 systemd[1]: Starting Ceph osd.6 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:47:26.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:47:26] "GET /metrics HTTP/1.1" 200 38461 "" "Prometheus/2.33.4" 2024-03-26T04:47:27.031 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 podman[65633]: 2024-03-26 04:47:26.677684972 +0000 UTC m=+0.018119109 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:47:27.032 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 podman[65633]: 2024-03-26 04:47:26.780098814 +0000 UTC m=+0.120532946 container create 13a3331df88bd0a405146f97a9b779a6934da89829233fe981f92ee5a56886ee (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T04:47:27.032 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 podman[65633]: 2024-03-26 04:47:26.957497414 +0000 UTC m=+0.297931542 container init 13a3331df88bd0a405146f97a9b779a6934da89829233fe981f92ee5a56886ee (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T04:47:27.032 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:26 smithi129 podman[65633]: 2024-03-26 04:47:26.960534644 +0000 UTC m=+0.300968777 container start 13a3331df88bd0a405146f97a9b779a6934da89829233fe981f92ee5a56886ee (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:47:27.032 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 podman[65633]: 2024-03-26 04:47:27.032536007 +0000 UTC m=+0.372970131 container attach 13a3331df88bd0a405146f97a9b779a6934da89829233fe981f92ee5a56886ee (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T04:47:27.719 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 bash[65633]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-26T04:47:27.720 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T04:47:27.720 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T04:47:27.720 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-26T04:47:27.720 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T04:47:27.720 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate[65648]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-26T04:47:27.720 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:27 smithi129 podman[65633]: 2024-03-26 04:47:27.423061645 +0000 UTC m=+0.763495770 container died 13a3331df88bd0a405146f97a9b779a6934da89829233fe981f92ee5a56886ee (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T04:47:28.269 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:28 smithi129 ceph-mon[57894]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:47:28.270 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 podman[65771]: 2024-03-26 04:47:28.036488036 +0000 UTC m=+0.604520054 container remove 13a3331df88bd0a405146f97a9b779a6934da89829233fe981f92ee5a56886ee (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T04:47:28.270 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 podman[65834]: 2024-03-26 04:47:28.151907316 +0000 UTC m=+0.017734152 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:47:28.270 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 podman[65834]: 2024-03-26 04:47:28.270813354 +0000 UTC m=+0.136640153 container create 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T04:47:28.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:28 smithi017 ceph-mon[64584]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:47:28.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:28 smithi017 ceph-mon[67142]: pgmap v180: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:47:28.529 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 podman[65834]: 2024-03-26 04:47:28.506710645 +0000 UTC m=+0.372537454 container init 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T04:47:28.529 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 podman[65834]: 2024-03-26 04:47:28.510012713 +0000 UTC m=+0.375839515 container start 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:47:28.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 ceph-osd[65852]: -- 172.21.15.129:0/1542042854 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b570a31e0 con 0x560b57074c00 2024-03-26T04:47:28.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 bash[65834]: 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e 2024-03-26T04:47:28.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:28 smithi129 systemd[1]: Started Ceph osd.6 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:47:29.394 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:47:29.055+0000 7f67b9c097c0 -1 Falling back to public interface 2024-03-26T04:47:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:29 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:29.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:29 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:29 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:29.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:29 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:29 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:29 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:30.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:30 smithi129 ceph-mon[57894]: pgmap v181: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:30.797 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:30 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T04:47:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:30 smithi017 ceph-mon[64584]: pgmap v181: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:30.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:30 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T04:47:30.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:30 smithi017 ceph-mon[67142]: pgmap v181: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:30.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:30 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded (PG_DEGRADED) 2024-03-26T04:47:31.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:47:31.755+0000 7f67b9c097c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-26T04:47:31.895 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:47:31.796+0000 7f67b9c097c0 -1 osd.6 136 log_to_monitors true 2024-03-26T04:47:32.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:32 smithi017 ceph-mon[64584]: pgmap v182: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:32.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:32 smithi017 ceph-mon[64584]: from='osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:47:32.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:32 smithi017 ceph-mon[64584]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:47:32.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:32 smithi017 ceph-mon[67142]: pgmap v182: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:32.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:32 smithi017 ceph-mon[67142]: from='osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:47:32.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:32 smithi017 ceph-mon[67142]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:47:32.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:32 smithi129 ceph-mon[57894]: pgmap v182: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:32.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:32 smithi129 ceph-mon[57894]: from='osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:47:32.981 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:32 smithi129 ceph-mon[57894]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:47:33.394 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:47:33.143+0000 7f67a7059700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:47:33.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:47:33.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: failed to encode map e139 with expected crc 2024-03-26T04:47:33.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: from='osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:47:33.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: osdmap e139: 8 total, 7 up, 8 in 2024-03-26T04:47:33.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:47:33.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: from='osd.6 ' entity='osd.6' 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: failed to encode map e139 with expected crc 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: from='osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: osdmap e139: 8 total, 7 up, 8 in 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: from='osd.6 ' entity='osd.6' 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:33.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:33 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: failed to encode map e139 with expected crc 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: from='osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: osdmap e139: 8 total, 7 up, 8 in 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: from='osd.6 ' entity='osd.6' 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:33 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: pgmap v184: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879] boot 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: pgmap v184: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879] boot 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.991 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: pgmap v184: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:34.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:47:34.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: osd.6 [v2:172.21.15.129:6816/528240879,v1:172.21.15.129:6817/528240879] boot 2024-03-26T04:47:34.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:47:34.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:47:34.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:34.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:35.902 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[67142]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[67142]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[67142]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[67142]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[64584]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[64584]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[64584]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:35 smithi017 ceph-mon[64584]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:47:35.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:35 smithi129 ceph-mon[57894]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:35 smithi129 ceph-mon[57894]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:35 smithi129 ceph-mon[57894]: failed to encode map e140 with expected crc 2024-03-26T04:47:35.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:35 smithi129 ceph-mon[57894]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:47:36.568 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:47:36.568 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 107s ago 11m 23.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:47:36.568 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (6m) 3s ago 11m 55.0M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:47:36.568 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (5m) 107s ago 10m 53.0M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:47:36.568 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (5m) 3s ago 15m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:47:36.568 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (5m) 107s ago 17m 507M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (5m) 107s ago 17m 49.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (4m) 3s ago 16m 38.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (4m) 107s ago 16m 38.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 107s ago 12m 15.8M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (6m) 3s ago 12m 19.1M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (3m) 107s ago 15m 47.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (3m) 107s ago 15m 44.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (2m) 107s ago 14m 50.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (112s) 107s ago 14m 12.5M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (77s) 3s ago 14m 56.0M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (42s) 3s ago 13m 45.6M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 0e80c6fa2668 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (8s) 3s ago 13m 12.1M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 8d9b6172909c 2024-03-26T04:47:36.569 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (12m) 3s ago 12m 65.6M 2992M 17.2.0 e1d6a67b021e 7c65b7838511 2024-03-26T04:47:36.570 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (5m) 3s ago 12m 72.1M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:47:36.570 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (10m) 107s ago 10m 73.5M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:47:36.570 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (10m) 3s ago 10m 77.8M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:47:36.570 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (10m) 107s ago 10m 74.0M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:47:36.570 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (10m) 3s ago 10m 75.5M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:47:36.682 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:47:36] "GET /metrics HTTP/1.1" 200 38461 "" "Prometheus/2.33.4" 2024-03-26T04:47:36.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:36 smithi129 ceph-mon[57894]: pgmap v186: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:36 smithi129 ceph-mon[57894]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:36 smithi129 ceph-mon[57894]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:36 smithi129 ceph-mon[57894]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[64584]: pgmap v186: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[64584]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[64584]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[64584]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[67142]: pgmap v186: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 82/726 objects degraded (11.295%) 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[67142]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[67142]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:36 smithi017 ceph-mon[67142]: failed to encode map e141 with expected crc 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:47:36.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 7 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 12 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:47:36.976 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:47:37.312 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "12/23 daemons upgraded", 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:47:37.313 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:47:37.725 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded 2024-03-26T04:47:37.725 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded 2024-03-26T04:47:37.725 INFO:teuthology.orchestra.run.smithi017.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-03-26T04:47:37.726 INFO:teuthology.orchestra.run.smithi017.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-03-26T04:47:37.847 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:37 smithi129 ceph-mon[57894]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.848 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:37 smithi129 ceph-mon[57894]: from='client.44431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.848 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:37 smithi129 ceph-mon[57894]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.848 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:37.848 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:37.848 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:37 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2498296666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:37.954 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[67142]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[67142]: from='client.44431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[67142]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2498296666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[64584]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[64584]: from='client.44431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[64584]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:37.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:37 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2498296666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.455 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:47:38.034+0000 7f879e59b700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: pgmap v188: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/726 objects degraded (11.295%) 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/94072982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:38.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: pgmap v188: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/726 objects degraded (11.295%) 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/94072982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:38.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:38.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:38.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:38.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:38.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:38 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: pgmap v188: 161 pgs: 16 active+undersized+degraded, 37 active+undersized, 108 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/726 objects degraded (11.295%) 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/94072982' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:39.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:47:39.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:38 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:39.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:39 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:39.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:39 smithi017 ceph-mon[64584]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:47:39.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:39 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:39.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:39 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:39.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:39 smithi017 ceph-mon[67142]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:47:39.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:39 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:40.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:39 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:39 smithi129 ceph-mon[57894]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-26T04:47:40.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:39 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:40.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:40 smithi017 ceph-mon[64584]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:47:40.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:40 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded) 2024-03-26T04:47:40.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:40 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:47:40.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:40 smithi017 ceph-mon[67142]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:47:40.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:40 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded) 2024-03-26T04:47:40.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:40 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:47:41.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:40 smithi129 ceph-mon[57894]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:47:41.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:40 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 82/726 objects degraded (11.295%), 16 pgs degraded) 2024-03-26T04:47:41.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:40 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:47:42.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:42 smithi017 ceph-mon[64584]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:47:42.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:42 smithi017 ceph-mon[67142]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:47:43.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:42 smithi129 ceph-mon[57894]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:47:44.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:44 smithi017 ceph-mon[64584]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2024-03-26T04:47:44.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:44 smithi017 ceph-mon[67142]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2024-03-26T04:47:45.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:44 smithi129 ceph-mon[57894]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 351 B/s rd, 0 op/s 2024-03-26T04:47:46.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:47:46] "GET /metrics HTTP/1.1" 200 38334 "" "Prometheus/2.33.4" 2024-03-26T04:47:46.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:46 smithi017 ceph-mon[64584]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-26T04:47:46.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:46 smithi017 ceph-mon[67142]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-26T04:47:47.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:46 smithi129 ceph-mon[57894]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-26T04:47:48.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:47 smithi129 ceph-mon[57894]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-03-26T04:47:48.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:47 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:47 smithi017 ceph-mon[67142]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-03-26T04:47:48.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:47 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:47 smithi017 ceph-mon[64584]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-03-26T04:47:48.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:47 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:49.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:48 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:49.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:48 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:49.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:48 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:50.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:49 smithi129 ceph-mon[57894]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:47:50.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:49 smithi017 ceph-mon[64584]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:47:50.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:49 smithi017 ceph-mon[67142]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:47:52.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:51 smithi129 ceph-mon[57894]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:51 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:52.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:51 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:51 smithi017 ceph-mon[64584]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:51 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:52.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:51 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:51 smithi017 ceph-mon[67142]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:51 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:52.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:51 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:54.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:53 smithi129 ceph-mon[57894]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:53 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:53 smithi129 ceph-mon[57894]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:54.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:53 smithi129 ceph-mon[57894]: Upgrade: osd.7 is safe to restart 2024-03-26T04:47:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[67142]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[67142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:54.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[67142]: Upgrade: osd.7 is safe to restart 2024-03-26T04:47:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[64584]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:54.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[64584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T04:47:54.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:53 smithi017 ceph-mon[64584]: Upgrade: osd.7 is safe to restart 2024-03-26T04:47:55.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:55 smithi129 ceph-mon[57894]: Upgrade: Updating osd.7 2024-03-26T04:47:55.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:55 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:55.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:55 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:47:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:55 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:55.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:55 smithi129 ceph-mon[57894]: Deploying daemon osd.7 on smithi129 2024-03-26T04:47:55.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[64584]: Upgrade: Updating osd.7 2024-03-26T04:47:55.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:55.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:47:55.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:55.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[64584]: Deploying daemon osd.7 on smithi129 2024-03-26T04:47:55.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[67142]: Upgrade: Updating osd.7 2024-03-26T04:47:55.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:47:55.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:47:55.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:55.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:55 smithi017 ceph-mon[67142]: Deploying daemon osd.7 on smithi129 2024-03-26T04:47:56.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:56 smithi017 ceph-mon[64584]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:56.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:56 smithi017 ceph-mon[67142]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:56.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:47:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:47:56] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.33.4" 2024-03-26T04:47:56.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:56 smithi129 ceph-mon[57894]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:57.894 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:57 smithi129 systemd[1]: Stopping Ceph osd.7 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:47:58.394 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[37864]: 2024-03-26T04:47:58.123+0000 7f3fbe64b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:47:58.394 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[37864]: 2024-03-26T04:47:58.123+0000 7f3fbe64b700 -1 osd.7 141 *** Got signal Terminated *** 2024-03-26T04:47:58.394 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[37864]: 2024-03-26T04:47:58.123+0000 7f3fbe64b700 -1 osd.7 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:47:58.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:58 smithi129 ceph-mon[57894]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:58.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:58 smithi129 ceph-mon[57894]: osd.7 marked itself down and dead 2024-03-26T04:47:58.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:58 smithi017 ceph-mon[64584]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:58.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:58 smithi017 ceph-mon[64584]: osd.7 marked itself down and dead 2024-03-26T04:47:58.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:58 smithi017 ceph-mon[67142]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:58.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:58 smithi017 ceph-mon[67142]: osd.7 marked itself down and dead 2024-03-26T04:47:59.382 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:59 smithi129 podman[67713]: 2024-03-26 04:47:59.110836456 +0000 UTC m=+1.191777021 container died 7c65b7838511e44ef9e093336cbdc1396355c9938b3961ef4b126aaca05eb97e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, vcs-type=git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, name=centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-26T04:47:59.632 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:59 smithi129 podman[67713]: 2024-03-26 04:47:59.498682096 +0000 UTC m=+1.579622654 container cleanup 7c65b7838511e44ef9e093336cbdc1396355c9938b3961ef4b126aaca05eb97e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, ceph=True, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=) 2024-03-26T04:47:59.632 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:59 smithi129 bash[67713]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7 2024-03-26T04:47:59.894 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:47:59 smithi129 podman[67741]: 2024-03-26 04:47:59.758554269 +0000 UTC m=+0.645849404 container remove 7c65b7838511e44ef9e093336cbdc1396355c9938b3961ef4b126aaca05eb97e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, release=754, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8) 2024-03-26T04:47:59.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:59 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:59 smithi129 ceph-mon[57894]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:47:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:59 smithi129 ceph-mon[57894]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T04:47:59.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:47:59 smithi129 ceph-mon[57894]: osdmap e142: 8 total, 7 up, 8 in 2024-03-26T04:47:59.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[64584]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[64584]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[64584]: osdmap e142: 8 total, 7 up, 8 in 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[67142]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[67142]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T04:47:59.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:47:59 smithi017 ceph-mon[67142]: osdmap e142: 8 total, 7 up, 8 in 2024-03-26T04:48:00.369 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:00 smithi129 podman[67835]: 2024-03-26 04:47:59.973468394 +0000 UTC m=+0.017768158 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:48:00.369 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:00 smithi129 podman[67835]: 2024-03-26 04:48:00.084822082 +0000 UTC m=+0.129121888 container create b8f79d07564220e1ebb29e77dbce4aaef583c1382062c52494ae16d2a979613d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T04:48:00.369 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:00 smithi129 podman[67835]: 2024-03-26 04:48:00.295507972 +0000 UTC m=+0.339807740 container init b8f79d07564220e1ebb29e77dbce4aaef583c1382062c52494ae16d2a979613d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T04:48:00.369 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:00 smithi129 podman[67835]: 2024-03-26 04:48:00.298904012 +0000 UTC m=+0.343203785 container start b8f79d07564220e1ebb29e77dbce4aaef583c1382062c52494ae16d2a979613d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T04:48:00.369 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:00 smithi129 podman[67835]: 2024-03-26 04:48:00.370372477 +0000 UTC m=+0.414672257 container attach b8f79d07564220e1ebb29e77dbce4aaef583c1382062c52494ae16d2a979613d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T04:48:00.633 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:00 smithi129 podman[67835]: 2024-03-26 04:48:00.591759939 +0000 UTC m=+0.636059705 container died b8f79d07564220e1ebb29e77dbce4aaef583c1382062c52494ae16d2a979613d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T04:48:00.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:00 smithi129 ceph-mon[57894]: pgmap v200: 161 pgs: 13 peering, 20 stale+active+clean, 128 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:00.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:00 smithi129 ceph-mon[57894]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T04:48:00.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:00 smithi129 ceph-mon[57894]: osdmap e143: 8 total, 7 up, 8 in 2024-03-26T04:48:00.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:00 smithi017 ceph-mon[64584]: pgmap v200: 161 pgs: 13 peering, 20 stale+active+clean, 128 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:00.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:00 smithi017 ceph-mon[64584]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T04:48:00.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:00 smithi017 ceph-mon[64584]: osdmap e143: 8 total, 7 up, 8 in 2024-03-26T04:48:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:00 smithi017 ceph-mon[67142]: pgmap v200: 161 pgs: 13 peering, 20 stale+active+clean, 128 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:00 smithi017 ceph-mon[67142]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-26T04:48:00.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:00 smithi017 ceph-mon[67142]: osdmap e143: 8 total, 7 up, 8 in 2024-03-26T04:48:01.615 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 podman[67861]: 2024-03-26 04:48:01.182689129 +0000 UTC m=+0.586007457 container remove b8f79d07564220e1ebb29e77dbce4aaef583c1382062c52494ae16d2a979613d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:48:01.615 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-26T04:48:01.616 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 systemd[1]: Stopped Ceph osd.7 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:48:01.616 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.7.service: Consumed 10.119s CPU time. 2024-03-26T04:48:01.616 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 systemd[1]: Starting Ceph osd.7 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:48:01.865 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 podman[67967]: 2024-03-26 04:48:01.519055762 +0000 UTC m=+0.018246951 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:48:01.865 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 podman[67967]: 2024-03-26 04:48:01.636065008 +0000 UTC m=+0.135256226 container create ae085e0bad2a5be509198af03fa8f9913e4e1055476a10508e9e3061f51fc5de (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T04:48:01.866 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 podman[67967]: 2024-03-26 04:48:01.863527434 +0000 UTC m=+0.362718607 container init ae085e0bad2a5be509198af03fa8f9913e4e1055476a10508e9e3061f51fc5de (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T04:48:01.866 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 podman[67967]: 2024-03-26 04:48:01.866953654 +0000 UTC m=+0.366144831 container start ae085e0bad2a5be509198af03fa8f9913e4e1055476a10508e9e3061f51fc5de (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T04:48:02.144 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:01 smithi129 podman[67967]: 2024-03-26 04:48:01.977252385 +0000 UTC m=+0.476443568 container attach ae085e0bad2a5be509198af03fa8f9913e4e1055476a10508e9e3061f51fc5de (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 bash[67967]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T04:48:02.637 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T04:48:02.638 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate[67982]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T04:48:02.638 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:02 smithi129 podman[67967]: 2024-03-26 04:48:02.335480405 +0000 UTC m=+0.834671580 container died ae085e0bad2a5be509198af03fa8f9913e4e1055476a10508e9e3061f51fc5de (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T04:48:02.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:02 smithi129 ceph-mon[57894]: pgmap v202: 161 pgs: 13 peering, 20 stale+active+clean, 128 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:02.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:02 smithi017 ceph-mon[64584]: pgmap v202: 161 pgs: 13 peering, 20 stale+active+clean, 128 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:02.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:02 smithi017 ceph-mon[67142]: pgmap v202: 161 pgs: 13 peering, 20 stale+active+clean, 128 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:48:03.373 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 podman[68099]: 2024-03-26 04:48:03.059407253 +0000 UTC m=+0.718425068 container remove ae085e0bad2a5be509198af03fa8f9913e4e1055476a10508e9e3061f51fc5de (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T04:48:03.373 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 podman[68163]: 2024-03-26 04:48:03.268663747 +0000 UTC m=+0.108125987 container create f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T04:48:03.373 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 podman[68163]: 2024-03-26 04:48:03.178595631 +0000 UTC m=+0.018057876 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:48:03.644 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 podman[68163]: 2024-03-26 04:48:03.504709652 +0000 UTC m=+0.344171890 container init f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T04:48:03.644 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 podman[68163]: 2024-03-26 04:48:03.508127531 +0000 UTC m=+0.347589768 container start f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T04:48:03.644 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 ceph-osd[68182]: -- 172.21.15.129:0/3213637445 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559486b571e0 con 0x559486b29400 2024-03-26T04:48:03.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:03 smithi017 ceph-mon[64584]: Health check failed: Degraded data redundancy: 1/726 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-03-26T04:48:03.954 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:03 smithi017 ceph-mon[67142]: Health check failed: Degraded data redundancy: 1/726 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-03-26T04:48:04.054 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 bash[68163]: f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 2024-03-26T04:48:04.054 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:03 smithi129 systemd[1]: Started Ceph osd.7 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:48:04.054 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:03 smithi129 ceph-mon[57894]: Health check failed: Degraded data redundancy: 1/726 objects degraded (0.138%), 1 pg degraded (PG_DEGRADED) 2024-03-26T04:48:04.394 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:04 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:48:04.053+0000 7f4198efb7c0 -1 Falling back to public interface 2024-03-26T04:48:04.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:04 smithi129 ceph-mon[57894]: pgmap v203: 161 pgs: 1 active+undersized+degraded, 13 peering, 15 stale+active+clean, 10 active+undersized, 122 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1/726 objects degraded (0.138%) 2024-03-26T04:48:04.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:04.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:04 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:05.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:04 smithi017 ceph-mon[67142]: pgmap v203: 161 pgs: 1 active+undersized+degraded, 13 peering, 15 stale+active+clean, 10 active+undersized, 122 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1/726 objects degraded (0.138%) 2024-03-26T04:48:05.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:05.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:04 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:05.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:04 smithi017 ceph-mon[64584]: pgmap v203: 161 pgs: 1 active+undersized+degraded, 13 peering, 15 stale+active+clean, 10 active+undersized, 122 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1/726 objects degraded (0.138%) 2024-03-26T04:48:05.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:05.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:04 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:06.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:05 smithi129 ceph-mon[57894]: pgmap v204: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 98/726 objects degraded (13.499%) 2024-03-26T04:48:06.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:05 smithi129 ceph-mon[57894]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-03-26T04:48:06.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:05 smithi017 ceph-mon[67142]: pgmap v204: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 98/726 objects degraded (13.499%) 2024-03-26T04:48:06.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:05 smithi017 ceph-mon[67142]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-03-26T04:48:06.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:05 smithi017 ceph-mon[64584]: pgmap v204: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 98/726 objects degraded (13.499%) 2024-03-26T04:48:06.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:05 smithi017 ceph-mon[64584]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-03-26T04:48:06.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:48:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:48:06] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.33.4" 2024-03-26T04:48:07.083 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:48:06.802+0000 7f4198efb7c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-26T04:48:07.083 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:06 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:48:06.836+0000 7f4198efb7c0 -1 osd.7 141 log_to_monitors true 2024-03-26T04:48:07.761 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:07 smithi129 ceph-mon[57894]: from='osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:48:07.761 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:07 smithi129 ceph-mon[57894]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:48:07.761 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:07.761 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:08.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[64584]: from='osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:48:08.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[64584]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:48:08.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:08.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:08.083 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[67142]: from='osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:48:08.084 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[67142]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:48:08.084 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:08.084 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:08.105 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:48:08.144 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:48:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:48:07.761+0000 7f418a353700 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:48:08.791 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:48:08.791 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (7m) 2m ago 12m 23.3M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (6m) 0s ago 11m 54.8M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (6m) 2m ago 10m 53.0M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (5m) 0s ago 16m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (6m) 2m ago 18m 507M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (5m) 2m ago 18m 49.5M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (5m) 0s ago 16m 38.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (5m) 2m ago 16m 38.8M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 2m ago 12m 15.8M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (6m) 0s ago 12m 22.0M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (4m) 2m ago 16m 47.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (3m) 2m ago 15m 44.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (2m) 2m ago 15m 50.4M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (2m) 2m ago 14m 12.5M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:48:08.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (109s) 0s ago 14m 57.1M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (75s) 0s ago 14m 46.5M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 0e80c6fa2668 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (40s) 0s ago 13m 49.3M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 8d9b6172909c 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (5s) 0s ago 13m 12.6M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 f505e855258d 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (6m) 0s ago 12m 72.2M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (11m) 2m ago 11m 73.5M - 17.2.0 e1d6a67b021e bf19becbd7ca 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (11m) 0s ago 11m 78.1M - 17.2.0 e1d6a67b021e 48386234b7c4 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (11m) 2m ago 11m 74.0M - 17.2.0 e1d6a67b021e 8599859214e9 2024-03-26T04:48:08.793 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (11m) 0s ago 11m 75.7M - 17.2.0 e1d6a67b021e bdb8a15497f8 2024-03-26T04:48:09.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: pgmap v205: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 98/726 objects degraded (13.499%) 2024-03-26T04:48:09.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:48:09.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: osdmap e144: 8 total, 7 up, 8 in 2024-03-26T04:48:09.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: from='osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:48:09.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:48:09.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:08 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.199 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:48:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:48:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:48:09.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 13 2024-03-26T04:48:09.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:48:09.201 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: pgmap v205: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 98/726 objects degraded (13.499%) 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: osdmap e144: 8 total, 7 up, 8 in 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: from='osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: pgmap v205: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 98/726 objects degraded (13.499%) 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: osdmap e144: 8 total, 7 up, 8 in 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: from='osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:08 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout: "osd", 2024-03-26T04:48:09.539 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:48:09.540 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:48:09.540 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:48:09.540 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "13/23 daemons upgraded", 2024-03-26T04:48:09.540 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T04:48:09.540 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:48:09.540 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:48:09.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='client.44467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788] boot 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='client.44479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1924224097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[64584]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:48:09.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='client.44467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788] boot 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='client.44479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1924224097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:09.956 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:09 smithi017 ceph-mon[67142]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 89/726 objects degraded (12.259%), 24 pgs degraded 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 89/726 objects degraded (12.259%), 24 pgs degraded 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-03-26T04:48:09.969 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-03-26T04:48:09.970 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.14 is active+undersized+degraded, acting [3,2] 2024-03-26T04:48:09.971 INFO:teuthology.orchestra.run.smithi017.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='client.44467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: osd.7 [v2:172.21.15.129:6824/1624597788,v1:172.21.15.129:6825/1624597788] boot 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='client.44479 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:10.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1924224097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:10.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:09 smithi129 ceph-mon[57894]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:48:11.066 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:10 smithi129 ceph-mon[57894]: pgmap v208: 161 pgs: 10 peering, 24 active+undersized+degraded, 35 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 89/726 objects degraded (12.259%) 2024-03-26T04:48:11.066 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:10 smithi129 ceph-mon[57894]: from='client.44491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:11.066 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:10 smithi129 ceph-mon[57894]: Health check update: Degraded data redundancy: 89/726 objects degraded (12.259%), 24 pgs degraded (PG_DEGRADED) 2024-03-26T04:48:11.066 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:10 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1461543200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:48:11.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[64584]: pgmap v208: 161 pgs: 10 peering, 24 active+undersized+degraded, 35 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 89/726 objects degraded (12.259%) 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[64584]: from='client.44491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[64584]: Health check update: Degraded data redundancy: 89/726 objects degraded (12.259%), 24 pgs degraded (PG_DEGRADED) 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1461543200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[67142]: pgmap v208: 161 pgs: 10 peering, 24 active+undersized+degraded, 35 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 89/726 objects degraded (12.259%) 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[67142]: from='client.44491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[67142]: Health check update: Degraded data redundancy: 89/726 objects degraded (12.259%), 24 pgs degraded (PG_DEGRADED) 2024-03-26T04:48:11.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:10 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1461543200' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:48:13.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: pgmap v210: 161 pgs: 10 peering, 24 active+undersized+degraded, 35 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/726 objects degraded (12.259%) 2024-03-26T04:48:13.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T04:48:13.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T04:48:13.146 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:12 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: pgmap v210: 161 pgs: 10 peering, 24 active+undersized+degraded, 35 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/726 objects degraded (12.259%) 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T04:48:13.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: pgmap v210: 161 pgs: 10 peering, 24 active+undersized+degraded, 35 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/726 objects degraded (12.259%) 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:13.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T04:48:13.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T04:48:13.209 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:12 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T04:48:14.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all osd 2024-03-26T04:48:14.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T04:48:14.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T04:48:14.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T04:48:14.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:48:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:14.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:13 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:14.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all osd 2024-03-26T04:48:14.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T04:48:14.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T04:48:14.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T04:48:14.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all osd 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:14.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:13 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:14.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:14 smithi017 ceph-mon[64584]: pgmap v211: 161 pgs: 10 peering, 14 active+undersized+degraded, 16 active+undersized, 121 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-03-26T04:48:14.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:14 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all mds 2024-03-26T04:48:14.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:14 smithi017 ceph-mon[67142]: pgmap v211: 161 pgs: 10 peering, 14 active+undersized+degraded, 16 active+undersized, 121 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-03-26T04:48:14.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:14 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all mds 2024-03-26T04:48:15.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:14 smithi129 ceph-mon[57894]: pgmap v211: 161 pgs: 10 peering, 14 active+undersized+degraded, 16 active+undersized, 121 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-03-26T04:48:15.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:14 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all mds 2024-03-26T04:48:16.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:48:16] "GET /metrics HTTP/1.1" 200 38555 "" "Prometheus/2.33.4" 2024-03-26T04:48:16.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: Upgrade: Updating rgw.foo.smithi017.yqadkl (1/4) 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: Deploying daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 14 pgs degraded) 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[64584]: Cluster is now healthy 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: Upgrade: Updating rgw.foo.smithi017.yqadkl (1/4) 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:16.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: Deploying daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:48:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 14 pgs degraded) 2024-03-26T04:48:16.706 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:16 smithi017 ceph-mon[67142]: Cluster is now healthy 2024-03-26T04:48:16.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-03-26T04:48:16.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: Upgrade: Updating rgw.foo.smithi017.yqadkl (1/4) 2024-03-26T04:48:16.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:16.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.yqadkl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:16.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: Deploying daemon rgw.foo.smithi017.yqadkl on smithi017 2024-03-26T04:48:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 14 pgs degraded) 2024-03-26T04:48:16.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:16 smithi129 ceph-mon[57894]: Cluster is now healthy 2024-03-26T04:48:18.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:18 smithi017 ceph-mon[67142]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:48:18.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:18 smithi017 ceph-mon[64584]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:48:18.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:18 smithi129 ceph-mon[57894]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:48:19.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:19 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:19.954 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:19 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:19.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:19 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:20.809 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:20 smithi017 ceph-mon[67142]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:48:20.809 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:20 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:20.809 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:20 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:20.810 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:20 smithi017 ceph-mon[64584]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:48:20.810 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:20 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:20.810 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:20 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:20.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:20 smithi129 ceph-mon[57894]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:48:20.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:20 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:20.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:20 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: Upgrade: Updating rgw.smpl.smithi017.jmwydb (2/4) 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: Deploying daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.592 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: Upgrade: Updating rgw.smpl.smithi017.jmwydb (2/4) 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: Deploying daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:22.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:48:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: Upgrade: Updating rgw.smpl.smithi017.jmwydb (2/4) 2024-03-26T04:48:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi017.jmwydb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:22.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: Deploying daemon rgw.smpl.smithi017.jmwydb on smithi017 2024-03-26T04:48:22.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:22.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:22.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:24.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:24 smithi017 ceph-mon[64584]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 31 op/s 2024-03-26T04:48:24.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:24 smithi017 ceph-mon[67142]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 31 op/s 2024-03-26T04:48:24.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:24 smithi129 ceph-mon[57894]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 31 op/s 2024-03-26T04:48:26.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:48:26] "GET /metrics HTTP/1.1" 200 38559 "" "Prometheus/2.33.4" 2024-03-26T04:48:27.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:26 smithi129 ceph-mon[57894]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 91 B/s wr, 133 op/s 2024-03-26T04:48:27.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:27.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:26 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:27.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-mon[64584]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 91 B/s wr, 133 op/s 2024-03-26T04:48:27.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:27.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:27.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-mon[67142]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 91 B/s wr, 133 op/s 2024-03-26T04:48:27.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:27.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:26 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:28.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:28 smithi129 ceph-mon[57894]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T04:48:28.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:28 smithi129 ceph-mon[57894]: Upgrade: Updating rgw.foo.smithi129.pgirvy (3/4) 2024-03-26T04:48:28.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:28.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:28.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:28 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:28.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:28 smithi129 ceph-mon[57894]: Deploying daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:48:28.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[64584]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T04:48:28.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[64584]: Upgrade: Updating rgw.foo.smithi129.pgirvy (3/4) 2024-03-26T04:48:28.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:28.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[64584]: Deploying daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[67142]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[67142]: Upgrade: Updating rgw.foo.smithi129.pgirvy (3/4) 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi129.pgirvy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:28.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:28 smithi017 ceph-mon[67142]: Deploying daemon rgw.foo.smithi129.pgirvy on smithi129 2024-03-26T04:48:29.790 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:29 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:29.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:29 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:29.955 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:29 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:30.775 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:30 smithi129 ceph-mon[57894]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:30.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:30 smithi017 ceph-mon[64584]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:30.954 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:30 smithi017 ceph-mon[67142]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:32.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:32 smithi129 ceph-mon[57894]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:32.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:32 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:32.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:32 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:32 smithi017 ceph-mon[64584]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:32 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:32.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:32 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:32.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:32 smithi017 ceph-mon[67142]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:32.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:32 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:32.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:32 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:34.307 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:34 smithi129 ceph-mon[57894]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 170 B/s wr, 274 op/s 2024-03-26T04:48:34.307 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:34 smithi129 ceph-mon[57894]: Upgrade: Updating rgw.smpl.smithi129.jvfyul (4/4) 2024-03-26T04:48:34.308 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:34.308 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:34.308 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:34 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:34.308 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:34 smithi129 ceph-mon[57894]: Deploying daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:48:34.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[64584]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 170 B/s wr, 274 op/s 2024-03-26T04:48:34.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[64584]: Upgrade: Updating rgw.smpl.smithi129.jvfyul (4/4) 2024-03-26T04:48:34.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:34.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:34.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[64584]: Deploying daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[67142]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 170 B/s wr, 274 op/s 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[67142]: Upgrade: Updating rgw.smpl.smithi129.jvfyul (4/4) 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi129.jvfyul", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:34.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:34 smithi017 ceph-mon[67142]: Deploying daemon rgw.smpl.smithi129.jvfyul on smithi129 2024-03-26T04:48:36.339 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:36 smithi129 ceph-mon[57894]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 219 KiB/s rd, 255 B/s wr, 346 op/s 2024-03-26T04:48:36.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:36 smithi017 ceph-mon[64584]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 219 KiB/s rd, 255 B/s wr, 346 op/s 2024-03-26T04:48:36.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:36 smithi017 ceph-mon[67142]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 219 KiB/s rd, 255 B/s wr, 346 op/s 2024-03-26T04:48:36.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:48:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:48:36] "GET /metrics HTTP/1.1" 200 38559 "" "Prometheus/2.33.4" 2024-03-26T04:48:38.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:38.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:38.051 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:38.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:38.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:38.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:38.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:38.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:38.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:39.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:38 smithi017 ceph-mon[64584]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:39.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:38 smithi017 ceph-mon[67142]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:39.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:38 smithi129 ceph-mon[57894]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:39.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:39 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:39.954 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:39 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:40.043 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:39 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:40.344 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (7m) 0s ago 12m 23.6M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (7m) 0s ago 12m 55.0M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 running (6m) 0s ago 11m 59.2M - 3.5 e1d6a67b021e 345d4453f506 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (6m) 0s ago 16m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (7m) 0s ago 18m 512M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (6m) 0s ago 18m 59.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (5m) 0s ago 17m 43.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (5m) 0s ago 17m 44.4M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 0s ago 13m 17.7M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:48:41.041 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (7m) 0s ago 13m 22.1M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (4m) 0s ago 16m 57.9M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (4m) 0s ago 16m 59.5M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (3m) 0s ago 15m 58.3M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (2m) 0s ago 15m 56.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (2m) 0s ago 15m 65.2M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (107s) 0s ago 14m 51.5M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 0e80c6fa2668 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (72s) 0s ago 14m 54.9M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 8d9b6172909c 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (37s) 0s ago 14m 51.1M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 f505e855258d 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (6m) 0s ago 13m 72.2M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (21s) 0s ago 11m 76.2M - 19.0.0-1775-g5e70124b 5f21c9264136 f46d4239aa2a 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (10s) 0s ago 11m 77.6M - 19.0.0-1775-g5e70124b 5f21c9264136 72a6bd224c22 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (15s) 0s ago 11m 78.7M - 19.0.0-1775-g5e70124b 5f21c9264136 56bd9495936a 2024-03-26T04:48:41.042 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (4s) 0s ago 11m 75.9M - 19.0.0-1775-g5e70124b 5f21c9264136 64d838d8c2a3 2024-03-26T04:48:41.043 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:40 smithi017 ceph-mon[64584]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 236 KiB/s rd, 255 B/s wr, 372 op/s 2024-03-26T04:48:41.043 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:41.043 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:40 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:41.043 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:40 smithi017 ceph-mon[67142]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 236 KiB/s rd, 255 B/s wr, 372 op/s 2024-03-26T04:48:41.043 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:41.043 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:40 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:41.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:40 smithi129 ceph-mon[57894]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 236 KiB/s rd, 255 B/s wr, 372 op/s 2024-03-26T04:48:41.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:41.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:40 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:41.456 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:48:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:48:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:48:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:48:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 17 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:48:41.457 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:48:41.811 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:48:41.811 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "osd", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "17/23 daemons upgraded", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading rgw daemons", 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-03-26T04:48:41.812 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:48:42.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/4231678319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:42.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:41 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/4231678319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/4231678319' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:41 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.245 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[64584]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[64584]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[64584]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2098915325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[67142]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[67142]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[67142]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.832 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.833 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.833 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:42 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2098915325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:48:42.893 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:42 smithi129 ceph-mon[57894]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:42 smithi129 ceph-mon[57894]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:42 smithi129 ceph-mon[57894]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:42 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:42 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:42.894 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:42 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2098915325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:48:45.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:44 smithi129 ceph-mon[57894]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:45.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:44 smithi017 ceph-mon[64584]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:45.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:44 smithi017 ceph-mon[67142]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T04:48:46.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 170 B/s wr, 223 op/s 2024-03-26T04:48:46.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: Detected new or changed devices on smithi129 2024-03-26T04:48:46.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.yqadkl"}]: dispatch 2024-03-26T04:48:46.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.yqadkl"}]': finished 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.pgirvy"}]: dispatch 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.pgirvy"}]': finished 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.jmwydb"}]: dispatch 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.jmwydb"}]': finished 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.jvfyul"}]: dispatch 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.jvfyul"}]': finished 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:46 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.653 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:48:46] "GET /metrics HTTP/1.1" 200 38550 "" "Prometheus/2.33.4" 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 170 B/s wr, 223 op/s 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: Detected new or changed devices on smithi129 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.654 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.yqadkl"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.yqadkl"}]': finished 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.pgirvy"}]: dispatch 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.pgirvy"}]': finished 2024-03-26T04:48:46.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.jmwydb"}]: dispatch 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.jmwydb"}]': finished 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.jvfyul"}]: dispatch 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.jvfyul"}]': finished 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 170 B/s wr, 223 op/s 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: Detected new or changed devices on smithi129 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.657 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.yqadkl"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.yqadkl"}]': finished 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.pgirvy"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi129.pgirvy"}]': finished 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.jmwydb"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi017.jmwydb"}]': finished 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.jvfyul"}]: dispatch 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi129.jvfyul"}]': finished 2024-03-26T04:48:46.658 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:46.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:48:46.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:46 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:47.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:47 smithi129 ceph-mon[57894]: Detected new or changed devices on smithi017 2024-03-26T04:48:47.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:47 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all rgw 2024-03-26T04:48:47.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:47 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T04:48:47.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:47 smithi129 ceph-mon[57894]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T04:48:47.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[64584]: Detected new or changed devices on smithi017 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all rgw 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[64584]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[67142]: Detected new or changed devices on smithi017 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all rgw 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T04:48:47.665 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:47 smithi017 ceph-mon[67142]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[67142]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[67142]: Upgrade: Updating iscsi.foo.smithi017.atsgyu 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[67142]: Deploying daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[64584]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T04:48:49.033 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[64584]: Upgrade: Updating iscsi.foo.smithi017.atsgyu 2024-03-26T04:48:49.034 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:49.034 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:48:49.034 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:49.034 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:48 smithi017 ceph-mon[64584]: Deploying daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:48:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:48 smithi129 ceph-mon[57894]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T04:48:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:48 smithi129 ceph-mon[57894]: Upgrade: Updating iscsi.foo.smithi017.atsgyu 2024-03-26T04:48:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:48:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.atsgyu", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:48:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:48 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:49.145 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:48 smithi129 ceph-mon[57894]: Deploying daemon iscsi.foo.smithi017.atsgyu on smithi017 2024-03-26T04:48:49.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:49 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:49.938 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:49 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:50.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:49 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:51.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:50 smithi129 ceph-mon[57894]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 125 op/s 2024-03-26T04:48:51.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:50 smithi017 ceph-mon[64584]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 125 op/s 2024-03-26T04:48:51.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:50 smithi017 ceph-mon[67142]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 125 op/s 2024-03-26T04:48:53.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:52 smithi129 ceph-mon[57894]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:53.144 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:53.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:52 smithi017 ceph-mon[64584]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:53.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:53.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:52 smithi017 ceph-mon[67142]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:53.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:55.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:54 smithi129 ceph-mon[57894]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:55.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:54 smithi017 ceph-mon[64584]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:55.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:54 smithi017 ceph-mon[67142]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:56.704 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:48:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:48:56] "GET /metrics HTTP/1.1" 200 38622 "" "Prometheus/2.33.4" 2024-03-26T04:48:57.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:56 smithi129 ceph-mon[57894]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:57.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:56 smithi017 ceph-mon[64584]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:57.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:56 smithi017 ceph-mon[67142]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:59.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:58 smithi129 ceph-mon[57894]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:59.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:58 smithi017 ceph-mon[64584]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:59.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:58 smithi017 ceph-mon[67142]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:00.045 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:48:59 smithi129 ceph-mon[57894]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:00.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:48:59 smithi017 ceph-mon[64584]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:00.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:48:59 smithi017 ceph-mon[67142]: from='client.25300 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:00.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:00 smithi017 ceph-mon[64584]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:00.954 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:00 smithi017 ceph-mon[67142]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:01.061 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:00 smithi129 ceph-mon[57894]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:01.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:01 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:49:01.395+0000 7f876ed73700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:49:02.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:02 smithi129 ceph-mon[57894]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:02.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:02 smithi129 ceph-mon[57894]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:49:02.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:02 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:02.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:02 smithi129 ceph-mon[57894]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T04:49:02.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:02 smithi129 ceph-mon[57894]: Health check failed: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-26T04:49:02.703 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[64584]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[64584]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[64584]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[64584]: Health check failed: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[67142]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[67142]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[67142]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T04:49:02.704 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:02 smithi017 ceph-mon[67142]: Health check failed: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-26T04:49:04.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:04 smithi017 ceph-mon[64584]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T04:49:04.704 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:04 smithi017 ceph-mon[67142]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T04:49:04.739 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:04 smithi129 ceph-mon[57894]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T04:49:06.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: Checking dashboard <-> RGW credentials 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:06.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.412 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: Checking dashboard <-> RGW credentials 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.413 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: Checking dashboard <-> RGW credentials 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:06.414 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:06.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:49:06] "GET /metrics HTTP/1.1" 200 38622 "" "Prometheus/2.33.4" 2024-03-26T04:49:07.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:07 smithi129 ceph-mon[57894]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-03-26T04:49:07.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:07 smithi129 ceph-mon[57894]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 0 op/s 2024-03-26T04:49:07.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:07 smithi129 ceph-mon[57894]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:49:07.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:07.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[64584]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-03-26T04:49:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[64584]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 0 op/s 2024-03-26T04:49:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[64584]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:49:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:07.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:07.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[67142]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-03-26T04:49:07.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[67142]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 147 B/s rd, 0 op/s 2024-03-26T04:49:07.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[67142]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:49:07.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:49:07.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:09.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:09 smithi129 ceph-mon[57894]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-26T04:49:09.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:09 smithi017 ceph-mon[64584]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-26T04:49:09.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:09 smithi017 ceph-mon[67142]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-03-26T04:49:11.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:11 smithi129 ceph-mon[57894]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T04:49:11.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:11 smithi017 ceph-mon[64584]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T04:49:11.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:11 smithi017 ceph-mon[67142]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T04:49:12.605 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:49:12.954 INFO:teuthology.orchestra.run.smithi017.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed." 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[64584]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/3003672309' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[64584]: from='client.34725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2686236722' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/186479986"}]: dispatch 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[67142]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/3003672309' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:49:13.244 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[67142]: from='client.34725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:13.245 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:13 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2686236722' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/186479986"}]: dispatch 2024-03-26T04:49:13.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:13 smithi129 ceph-mon[57894]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T04:49:13.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:13 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/3003672309' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:49:13.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:13 smithi129 ceph-mon[57894]: from='client.34725 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:13.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:13 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2686236722' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/186479986"}]: dispatch 2024-03-26T04:49:13.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch ps' 2024-03-26T04:49:14.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[64584]: from='client.44695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:14.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2686236722' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/186479986"}]': finished 2024-03-26T04:49:14.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[64584]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:49:14.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/406419707' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3990286760"}]: dispatch 2024-03-26T04:49:14.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[67142]: from='client.44695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:14.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2686236722' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/186479986"}]': finished 2024-03-26T04:49:14.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[67142]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:49:14.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:14 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/406419707' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3990286760"}]: dispatch 2024-03-26T04:49:14.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:14 smithi129 ceph-mon[57894]: from='client.44695 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:14.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:14 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2686236722' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/186479986"}]': finished 2024-03-26T04:49:14.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:14 smithi129 ceph-mon[57894]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:49:14.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:14 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/406419707' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3990286760"}]: dispatch 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[64584]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/406419707' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3990286760"}]': finished 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[64584]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2183918160' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]: dispatch 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[64584]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]: dispatch 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[67142]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-26T04:49:15.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/406419707' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3990286760"}]': finished 2024-03-26T04:49:15.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[67142]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:49:15.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2183918160' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]: dispatch 2024-03-26T04:49:15.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:15 smithi017 ceph-mon[67142]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]: dispatch 2024-03-26T04:49:15.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:15 smithi129 ceph-mon[57894]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 3 op/s 2024-03-26T04:49:15.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:15 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/406419707' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3990286760"}]': finished 2024-03-26T04:49:15.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:15 smithi129 ceph-mon[57894]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:49:15.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:15 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2183918160' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]: dispatch 2024-03-26T04:49:15.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:15 smithi129 ceph-mon[57894]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]: dispatch 2024-03-26T04:49:15.767 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (8m) 10s ago 13m 23.6M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (7m) 10s ago 12m 53.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 unknown 10s ago 12m - - 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (7m) 10s ago 17m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (7m) 10s ago 19m 514M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (6m) 10s ago 19m 58.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (6m) 10s ago 17m 41.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (6m) 10s ago 17m 45.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (8m) 10s ago 13m 15.9M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (7m) 10s ago 13m 19.0M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (5m) 10s ago 17m 58.0M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (4m) 10s ago 16m 58.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (4m) 10s ago 16m 58.8M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:49:15.768 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (3m) 10s ago 16m 56.8M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (2m) 10s ago 15m 64.3M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (2m) 10s ago 15m 50.6M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 0e80c6fa2668 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (107s) 10s ago 14m 55.0M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 8d9b6172909c 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (72s) 10s ago 14m 50.3M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 f505e855258d 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (7m) 10s ago 13m 72.2M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (56s) 10s ago 12m 76.5M - 19.0.0-1775-g5e70124b 5f21c9264136 f46d4239aa2a 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (44s) 10s ago 12m 77.9M - 19.0.0-1775-g5e70124b 5f21c9264136 72a6bd224c22 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (50s) 10s ago 12m 79.0M - 19.0.0-1775-g5e70124b 5f21c9264136 56bd9495936a 2024-03-26T04:49:15.769 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (39s) 10s ago 12m 76.2M - 19.0.0-1775-g5e70124b 5f21c9264136 64d838d8c2a3 2024-03-26T04:49:16.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions' 2024-03-26T04:49:16.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:16 smithi129 ceph-mon[57894]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]': finished 2024-03-26T04:49:16.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:16 smithi129 ceph-mon[57894]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:49:16.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:16 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/162407831' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2479407874"}]: dispatch 2024-03-26T04:49:16.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:16 smithi129 ceph-mon[57894]: from='client.44719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:16.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[64584]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]': finished 2024-03-26T04:49:16.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[64584]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:49:16.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/162407831' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2479407874"}]: dispatch 2024-03-26T04:49:16.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[64584]: from='client.44719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:16.410 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[67142]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2465339214"}]': finished 2024-03-26T04:49:16.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[67142]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:49:16.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/162407831' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2479407874"}]: dispatch 2024-03-26T04:49:16.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-mon[67142]: from='client.44719 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:49:16.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:49:16] "GET /metrics HTTP/1.1" 200 38700 "" "Prometheus/2.33.4" 2024-03-26T04:49:17.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:17 smithi129 ceph-mon[57894]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T04:49:17.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:17 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/162407831' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2479407874"}]': finished 2024-03-26T04:49:17.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:17 smithi129 ceph-mon[57894]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:49:17.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:17 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1793404364' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/510486991"}]: dispatch 2024-03-26T04:49:17.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[67142]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T04:49:17.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/162407831' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2479407874"}]': finished 2024-03-26T04:49:17.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[67142]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:49:17.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1793404364' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/510486991"}]: dispatch 2024-03-26T04:49:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[64584]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T04:49:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/162407831' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2479407874"}]': finished 2024-03-26T04:49:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[64584]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:49:17.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:17 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1793404364' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/510486991"}]: dispatch 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1793404364' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/510486991"}]': finished 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[67142]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2340277947' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]: dispatch 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[67142]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]: dispatch 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1793404364' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/510486991"}]': finished 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[64584]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2340277947' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]: dispatch 2024-03-26T04:49:18.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:18 smithi017 ceph-mon[64584]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]: dispatch 2024-03-26T04:49:18.350 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:49:18.351 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 17 2024-03-26T04:49:18.352 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:49:18.352 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:49:18.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:18 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1793404364' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/510486991"}]': finished 2024-03-26T04:49:18.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:18 smithi129 ceph-mon[57894]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:49:18.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:18 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2340277947' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]: dispatch 2024-03-26T04:49:18.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:18 smithi129 ceph-mon[57894]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]: dispatch 2024-03-26T04:49:18.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-26T04:49:19.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[64584]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[64584]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]': finished 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[64584]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/2678744349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[67142]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[67142]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]': finished 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[67142]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:49:19.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:19 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/2678744349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:49:19.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:19 smithi129 ceph-mon[57894]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:49:19.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:19 smithi129 ceph-mon[57894]: from='client.? ' entity='client.iscsi.foo.smithi017.atsgyu' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/186479986"}]': finished 2024-03-26T04:49:19.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:19 smithi129 ceph-mon[57894]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:49:19.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:19 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/2678744349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:49:20.778 INFO:teuthology.orchestra.run.smithi017.stdout:wait for servicemap items w/ changing names to refresh 2024-03-26T04:49:21.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:21 smithi017 ceph-mon[64584]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:49:21.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:21 smithi017 ceph-mon[67142]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:49:21.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:21 smithi129 ceph-mon[57894]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:49:21.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'sleep 60' 2024-03-26T04:49:22.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:22.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:22.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:23.125 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:23 smithi017 ceph-mon[64584]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:23.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:23 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:23.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:23 smithi017 ceph-mon[67142]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:23.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:23 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:23.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:23 smithi129 ceph-mon[57894]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:23.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:23 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:25.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:25 smithi129 ceph-mon[57894]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:49:25.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:25 smithi017 ceph-mon[64584]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:49:25.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:25 smithi017 ceph-mon[67142]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:49:26.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:49:26] "GET /metrics HTTP/1.1" 200 38701 "" "Prometheus/2.33.4" 2024-03-26T04:49:27.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:27 smithi129 ceph-mon[57894]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:49:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:27 smithi017 ceph-mon[64584]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:49:27.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:27 smithi017 ceph-mon[67142]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:49:29.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:29 smithi129 ceph-mon[57894]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:49:29.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:29 smithi017 ceph-mon[64584]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:49:29.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:29 smithi017 ceph-mon[67142]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:49:31.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:31 smithi129 ceph-mon[57894]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:49:31.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:31 smithi017 ceph-mon[64584]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:49:31.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:31 smithi017 ceph-mon[67142]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:49:33.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:33 smithi129 ceph-mon[57894]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:33.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:33 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:33.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:33 smithi017 ceph-mon[64584]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:33.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:33 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:33.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:33 smithi017 ceph-mon[67142]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:33.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:33 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:35.393 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:35 smithi129 ceph-mon[57894]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:35.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:35 smithi017 ceph-mon[64584]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:35.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:35 smithi017 ceph-mon[67142]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:36.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:49:36] "GET /metrics HTTP/1.1" 200 38701 "" "Prometheus/2.33.4" 2024-03-26T04:49:37.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:37 smithi017 ceph-mon[64584]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:37.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:37.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:37 smithi017 ceph-mon[67142]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:37.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:37.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:37 smithi129 ceph-mon[57894]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:37.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:39.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:39 smithi017 ceph-mon[64584]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:39.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:39 smithi017 ceph-mon[67142]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:39.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:39 smithi129 ceph-mon[57894]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:41.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:41 smithi017 ceph-mon[64584]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:41.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:41 smithi017 ceph-mon[67142]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:41.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:41 smithi129 ceph-mon[57894]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:43.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:43 smithi017 ceph-mon[64584]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:43.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:43 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:43.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:43 smithi017 ceph-mon[67142]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:43.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:43 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:43.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:43 smithi129 ceph-mon[57894]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:43.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:43 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:45.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:45 smithi017 ceph-mon[64584]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:45.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:45 smithi017 ceph-mon[67142]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:45.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:45 smithi129 ceph-mon[57894]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:46.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:49:46] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.33.4" 2024-03-26T04:49:47.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:47 smithi017 ceph-mon[64584]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:47.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:47 smithi017 ceph-mon[67142]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:47.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:47 smithi129 ceph-mon[57894]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:49.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:49 smithi017 ceph-mon[64584]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:49.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:49 smithi017 ceph-mon[67142]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:49.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:49 smithi129 ceph-mon[57894]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:51.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:51 smithi017 ceph-mon[64584]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:51 smithi017 ceph-mon[67142]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:51.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:51 smithi129 ceph-mon[57894]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:52.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:52.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:52.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:53.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:53 smithi017 ceph-mon[64584]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:53.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:53 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:53.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:53 smithi017 ceph-mon[67142]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:53.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:53 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:53.642 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:53 smithi129 ceph-mon[57894]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:53.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:53 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:55.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:55 smithi017 ceph-mon[64584]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:55.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:55 smithi017 ceph-mon[67142]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:55.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:55 smithi129 ceph-mon[57894]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:56.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:49:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:49:56] "GET /metrics HTTP/1.1" 200 38699 "" "Prometheus/2.33.4" 2024-03-26T04:49:57.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:57 smithi017 ceph-mon[64584]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:57.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:57 smithi017 ceph-mon[67142]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:57.642 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:57 smithi129 ceph-mon[57894]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:59.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:49:59 smithi017 ceph-mon[64584]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:59.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:49:59 smithi017 ceph-mon[67142]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:59.642 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:49:59 smithi129 ceph-mon[57894]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:00.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:00.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Upgrade daemon: iscsi.foo.smithi017.atsgyu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Creating ceph-iscsi config... 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d67a15696\x2deb29\x2d11ee\x2d95d0\x2d87774f69a715.slice/ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Non-zero exit code 1 from systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: Traceback (most recent call last): 2024-03-26T04:50:00.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: return _run_code(code, main_globals, None, 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: exec(code, run_globals) 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1029, in deploy_daemon 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1165, in deploy_daemon_units 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: RuntimeError: Failed command: systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:00.455 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[64584]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Upgrade daemon: iscsi.foo.smithi017.atsgyu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Creating ceph-iscsi config... 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d67a15696\x2deb29\x2d11ee\x2d95d0\x2d87774f69a715.slice/ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service 2024-03-26T04:50:00.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Non-zero exit code 1 from systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: Traceback (most recent call last): 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: return _run_code(code, main_globals, None, 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: exec(code, run_globals) 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:50:00.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:50:00.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:50:00.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1029, in deploy_daemon 2024-03-26T04:50:00.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1165, in deploy_daemon_units 2024-03-26T04:50:00.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:50:00.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: RuntimeError: Failed command: systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:00.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:00 smithi017 ceph-mon[67142]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Upgrade daemon: iscsi.foo.smithi017.atsgyu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Creating ceph-iscsi config... 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:50:00.643 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d67a15696\x2deb29\x2d11ee\x2d95d0\x2d87774f69a715.slice/ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Non-zero exit code 1 from systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: Traceback (most recent call last): 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: return _run_code(code, main_globals, None, 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: exec(code, run_globals) 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:50:00.644 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1029, in deploy_daemon 2024-03-26T04:50:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1165, in deploy_daemon_units 2024-03-26T04:50:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:50:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: RuntimeError: Failed command: systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:00.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:00 smithi129 ceph-mon[57894]: See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:01.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:01 smithi017 ceph-mon[64584]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:01.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:01 smithi017 ceph-mon[67142]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:01.544 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:01 smithi129 ceph-mon[57894]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:04.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:03 smithi129 ceph-mon[57894]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:04.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:03 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:04.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:03 smithi017 ceph-mon[64584]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:04.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:03 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:04.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:03 smithi017 ceph-mon[67142]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:04.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:03 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:06.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:05 smithi129 ceph-mon[57894]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:06.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:05 smithi017 ceph-mon[64584]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:06.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:05 smithi017 ceph-mon[67142]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:06.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:50:06] "GET /metrics HTTP/1.1" 200 38699 "" "Prometheus/2.33.4" 2024-03-26T04:50:07.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:07.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:07.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:06 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:50:07.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:07.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:07.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:50:07.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:07.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:07.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:06 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' 2024-03-26T04:50:08.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:07 smithi129 ceph-mon[57894]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:08.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:07 smithi129 ceph-mon[57894]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-26T04:50:08.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:07 smithi129 ceph-mon[57894]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:08.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:07 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[64584]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[64584]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[64584]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[67142]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[67142]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[67142]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:08.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:07 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:10.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:09 smithi129 ceph-mon[57894]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:09 smithi017 ceph-mon[64584]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:09 smithi017 ceph-mon[67142]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:12.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:11 smithi129 ceph-mon[57894]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:12.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:11 smithi017 ceph-mon[64584]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:12.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:11 smithi017 ceph-mon[67142]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:14.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:13 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:13 smithi129 ceph-mon[57894]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T04:50:14.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:13 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:13 smithi017 ceph-mon[64584]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T04:50:14.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:13 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:13 smithi017 ceph-mon[67142]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T04:50:16.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:15 smithi129 ceph-mon[57894]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:16.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:15 smithi017 ceph-mon[64584]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:16.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:15 smithi017 ceph-mon[67142]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:16.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:16 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:50:16] "GET /metrics HTTP/1.1" 200 38703 "" "Prometheus/2.33.4" 2024-03-26T04:50:18.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:17 smithi129 ceph-mon[57894]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:18.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:17 smithi017 ceph-mon[64584]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:18.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:17 smithi017 ceph-mon[67142]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:20.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:19 smithi129 ceph-mon[57894]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:20.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:19 smithi017 ceph-mon[67142]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:20.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:19 smithi017 ceph-mon[64584]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:22.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:21 smithi129 ceph-mon[57894]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:22.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:21 smithi017 ceph-mon[64584]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:22.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:21 smithi017 ceph-mon[67142]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:23.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:22 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:23.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:22 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:23.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:22 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:23.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch ps' 2024-03-26T04:50:24.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:23 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:23 smithi017 ceph-mon[64584]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:24.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:23 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:23 smithi017 ceph-mon[67142]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:24.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:23 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:23 smithi129 ceph-mon[57894]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:25.824 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (9m) 80s ago 14m 23.6M - 0.23.0 ba2b418f427c 0c94534064e5 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi129 *:3000 running (8m) 80s ago 14m 53.7M - 8.3.5 dad864ee21e9 45c2baa57c4d 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.atsgyu smithi017 unknown 80s ago 13m - - 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi129 *:8443,9283,8765 running (8m) 80s ago 18m 451M - 19.0.0-1775-g5e70124b 5f21c9264136 7a621834387f 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (8m) 80s ago 20m 514M - 19.0.0-1775-g5e70124b 5f21c9264136 4836c6142fed 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (7m) 80s ago 20m 58.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 c90cfaf45cc1 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi129 running (7m) 80s ago 18m 41.3M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 fe20e3dd0032 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (7m) 80s ago 19m 45.2M 2048M 19.0.0-1775-g5e70124b 5f21c9264136 2c7ea6faebf3 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (9m) 80s ago 15m 15.9M - 1.3.1 1dbe0e931976 d9bfcddb7433 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi129 *:9100 running (9m) 80s ago 15m 19.0M - 1.3.1 1dbe0e931976 1f9a7c9aca8f 2024-03-26T04:50:25.825 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (6m) 80s ago 18m 58.0M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 f4afeb9d64a6 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (5m) 80s ago 18m 58.7M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 c04a08104bf3 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (5m) 80s ago 17m 58.8M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 53714135178c 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (4m) 80s ago 17m 56.8M 2737M 19.0.0-1775-g5e70124b 5f21c9264136 61ddd8e4617e 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi129 running (4m) 80s ago 16m 64.3M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 cc81d25035e9 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi129 running (3m) 80s ago 16m 50.6M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 0e80c6fa2668 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi129 running (2m) 80s ago 16m 55.0M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 8d9b6172909c 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi129 running (2m) 80s ago 15m 50.3M 2992M 19.0.0-1775-g5e70124b 5f21c9264136 f505e855258d 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi129 *:9095 running (8m) 80s ago 15m 72.2M - 2.33.4 514e6a882f6e 807223168a43 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.yqadkl smithi017 *:8000 running (2m) 80s ago 13m 76.5M - 19.0.0-1775-g5e70124b 5f21c9264136 f46d4239aa2a 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi129.pgirvy smithi129 *:8000 running (114s) 80s ago 13m 77.9M - 19.0.0-1775-g5e70124b 5f21c9264136 72a6bd224c22 2024-03-26T04:50:25.826 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi017.jmwydb smithi017 *:80 running (2m) 80s ago 13m 79.0M - 19.0.0-1775-g5e70124b 5f21c9264136 56bd9495936a 2024-03-26T04:50:25.827 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.smpl.smithi129.jvfyul smithi129 *:80 running (109s) 80s ago 13m 76.2M - 19.0.0-1775-g5e70124b 5f21c9264136 64d838d8c2a3 2024-03-26T04:50:26.078 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:25 smithi017 ceph-mon[64584]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:26.078 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:25 smithi017 ceph-mon[67142]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:26.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:25 smithi129 ceph-mon[57894]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:26.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions' 2024-03-26T04:50:26.684 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:26 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:50:26] "GET /metrics HTTP/1.1" 200 38698 "" "Prometheus/2.33.4" 2024-03-26T04:50:26.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:26 smithi017 ceph-mon[64584]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:26.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:26 smithi017 ceph-mon[67142]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:27.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:26 smithi129 ceph-mon[57894]: from='client.34785 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:27.977 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:27 smithi017 ceph-mon[64584]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:27.977 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:27 smithi017 ceph-mon[67142]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:28.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:27 smithi129 ceph-mon[57894]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:28.512 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:50:28.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-03-26T04:50:28.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 3 2024-03-26T04:50:28.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:50:28.512 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-03-26T04:50:28.512 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 2 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 8 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 4 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)": 17 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-03-26T04:50:28.513 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:50:29.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:28 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1950694348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:29.117 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:28 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1950694348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:29.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch upgrade status' 2024-03-26T04:50:29.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:28 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1950694348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:30.076 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:29 smithi017 ceph-mon[64584]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:30.076 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:29 smithi017 ceph-mon[67142]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:30.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:29 smithi129 ceph-mon[57894]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:31.064 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:50:31.064 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a", 2024-03-26T04:50:31.064 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-03-26T04:50:31.064 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T04:50:31.064 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "osd", 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw", 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "17/23 daemons upgraded", 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed.", 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": true 2024-03-26T04:50:31.065 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:50:31.667 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph health detail' 2024-03-26T04:50:31.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:31 smithi017 ceph-mon[64584]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:31.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:31 smithi017 ceph-mon[67142]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:32.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:31 smithi129 ceph-mon[57894]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:33.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:32 smithi017 ceph-mon[64584]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:33.135 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:32 smithi017 ceph-mon[67142]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:33.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:32 smithi129 ceph-mon[57894]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:33.630 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:33.630 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.atsgyu on smithi017 is in unknown state 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.atsgyu on host smithi017 failed. 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrade daemon: iscsi.foo.smithi017.atsgyu: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi017.atsgyu ... 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:Creating ceph-iscsi config... 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/iscsi-gateway.cfg 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:Write file: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/iscsi.foo.smithi017.atsgyu/tcmu-runner-entrypoint.sh 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d67a15696\x2deb29\x2d11ee\x2d95d0\x2d87774f69a715.slice/ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout:Traceback (most recent call last): 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout: return _run_code(code, main_globals, None, 2024-03-26T04:50:33.631 INFO:teuthology.orchestra.run.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: exec(code, run_globals) 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5520, in 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 5508, in main 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3038, in command_deploy_from 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3073, in _common_deploy 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 3093, in _deploy_daemon_container 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1029, in deploy_daemon 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/__main__.py", line 1165, in deploy_daemon_units 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/cephadm.3901b145d41c00490e411ad819563b03199fcd135b8259c36bb7a3e597c7f206/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout:RuntimeError: Failed command: systemctl start ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu: Job for ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service failed because the control process exited with error code. 2024-03-26T04:50:33.632 INFO:teuthology.orchestra.run.smithi017.stdout:See "systemctl status ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" and "journalctl -xeu ceph-67a15696-eb29-11ee-95d0-87774f69a715@iscsi.foo.smithi017.atsgyu.service" for details. 2024-03-26T04:50:33.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:33 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:33.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:33 smithi017 ceph-mon[64584]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:33.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:33 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/3369222463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:50:33.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:33 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:33.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:33 smithi017 ceph-mon[67142]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:33.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:33 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/3369222463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:50:34.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:33 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:34.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:33 smithi129 ceph-mon[57894]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:34.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:33 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/3369222463' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T04:50:34.340 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-26T04:50:35.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:35 smithi017 ceph-mon[64584]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:35.861 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:35 smithi017 ceph-mon[67142]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:36.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:35 smithi129 ceph-mon[57894]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:36.390 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-03-26T04:50:36.681 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:36 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:50:36] "GET /metrics HTTP/1.1" 200 38698 "" "Prometheus/2.33.4" 2024-03-26T04:50:36.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:36 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/348630608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:36.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:36 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/348630608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:37.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-26T04:50:37.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:36 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/348630608' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:37.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:37 smithi017 ceph-mon[64584]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:37.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:37 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:37.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:37 smithi017 ceph-mon[67142]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:37.953 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:37 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:38.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:37 smithi129 ceph-mon[57894]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:38.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:37 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:39.193 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)" 2024-03-26T04:50:39.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:39 smithi017 ceph-mon[64584]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:39.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:39 smithi017 ceph-mon[64584]: from='client.? 172.21.15.17:0/1627757218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:39.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:39 smithi017 ceph-mon[67142]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:39.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:39 smithi017 ceph-mon[67142]: from='client.? 172.21.15.17:0/1627757218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:39.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-26T04:50:40.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:39 smithi129 ceph-mon[57894]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:40.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:39 smithi129 ceph-mon[57894]: from='client.? 172.21.15.17:0/1627757218' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T04:50:42.084 INFO:teuthology.orchestra.run.smithi017.stdout:osd 8 97s ago - 2024-03-26T04:50:42.084 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:41 smithi017 ceph-mon[64584]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:42.085 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:41 smithi017 ceph-mon[67142]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:42.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:41 smithi129 ceph-mon[57894]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:42.671 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T04:50:42.681 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-03-26T04:50:42.681 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-26T04:50:44.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:43 smithi129 ceph-mon[57894]: from='client.44779 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:44.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:43 smithi129 ceph-mon[57894]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:43 smithi129 ceph-mon[57894]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:44.156 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:43 smithi017 ceph-mon[67142]: from='client.44779 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:44.156 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:43 smithi017 ceph-mon[67142]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.156 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:43 smithi017 ceph-mon[67142]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:44.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:43 smithi017 ceph-mon[64584]: from='client.44779 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:44.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:43 smithi017 ceph-mon[64584]: from='client.54509 -' entity='client.iscsi.foo.smithi017.atsgyu' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.156 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:43 smithi017 ceph-mon[64584]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:45.258 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-03-26T04:50:45.258 INFO:teuthology.orchestra.run.smithi017.stdout: "image": "quay.io/ceph/ceph", 2024-03-26T04:50:45.258 INFO:teuthology.orchestra.run.smithi017.stdout: "registry": "quay.io", 2024-03-26T04:50:45.258 INFO:teuthology.orchestra.run.smithi017.stdout: "bare_image": "ceph/ceph", 2024-03-26T04:50:45.258 INFO:teuthology.orchestra.run.smithi017.stdout: "versions": [] 2024-03-26T04:50:45.258 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-03-26T04:50:45.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:45 smithi017 ceph-mon[64584]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:45.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:45 smithi017 ceph-mon[64584]: from='client.34827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:45.851 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:45 smithi017 ceph-mon[67142]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:45.852 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:45 smithi017 ceph-mon[67142]: from='client.34827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:45.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-26T04:50:46.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:45 smithi129 ceph-mon[57894]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:46.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:45 smithi129 ceph-mon[57894]: from='client.34827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:46.452 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:46 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:50:46] "GET /metrics HTTP/1.1" 200 38701 "" "Prometheus/2.33.4" 2024-03-26T04:50:48.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:47 smithi129 ceph-mon[57894]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:48.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:47 smithi017 ceph-mon[64584]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:48.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:47 smithi017 ceph-mon[67142]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:48.563 INFO:teuthology.orchestra.run.smithi017.stdout: "16.2.0", 2024-03-26T04:50:49.144 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 67a15696-eb29-11ee-95d0-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-26T04:50:50.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:49 smithi129 ceph-mon[57894]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:50.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:49 smithi129 ceph-mon[57894]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:50.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:49 smithi017 ceph-mon[67142]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:50.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:49 smithi017 ceph-mon[67142]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:50.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:49 smithi017 ceph-mon[64584]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:50.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:49 smithi017 ceph-mon[64584]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "v16.2.2", 2024-03-26T04:50:51.819 INFO:teuthology.orchestra.run.smithi017.stdout: "v16.2.2-20210505", 2024-03-26T04:50:52.133 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:51 smithi017 ceph-mon[64584]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:52.133 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:51 smithi017 ceph-mon[67142]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:52.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:51 smithi129 ceph-mon[57894]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:52.465 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T04:50:52.476 INFO:tasks.cephadm:Teardown begin 2024-03-26T04:50:52.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:50:52.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:50:52.541 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T04:50:52.542 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:50:52.561 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:50:52.599 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T04:50:52.599 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T04:50:52.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:50:52.702 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:52 smithi017 systemd[1]: Stopping Ceph mon.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:50:53.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:52 smithi129 ceph-mon[57894]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:53.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:52 smithi129 ceph-mon[57894]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:53.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-mon[67142]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:53.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-mon[67142]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-mon[64584]: from='client.34839 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-mon[64584]: from='mgr.34107 172.21.15.17:0/2447405542' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a[64580]: 2024-03-26T04:50:52.905+0000 7fb9ab986700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a[64580]: 2024-03-26T04:50:52.905+0000 7fb9ab986700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:52 smithi017 podman[91250]: 2024-03-26 04:50:52.926641238 +0000 UTC m=+0.130384377 container died c90cfaf45cc1388c245e63ba39466e875010a898df1edee57130cf18024e6b3d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:50:52] ENGINE Bus STOPPING 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:50:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:50:53.203 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:50:52] ENGINE Bus STOPPED 2024-03-26T04:50:53.204 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:52 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:50:52] ENGINE Bus STARTING 2024-03-26T04:50:53.204 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: CherryPy Checker: 2024-03-26T04:50:53.204 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: The Application mounted at '' has an empty config. 2024-03-26T04:50:53.204 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: 2024-03-26T04:50:53.204 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:50:53] ENGINE Serving on http://:::9283 2024-03-26T04:50:53.204 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: [26/Mar/2024:04:50:53] ENGINE Bus STARTED 2024-03-26T04:50:53.529 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:53 smithi017 podman[91250]: 2024-03-26 04:50:53.323340878 +0000 UTC m=+0.527084060 container cleanup c90cfaf45cc1388c245e63ba39466e875010a898df1edee57130cf18024e6b3d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T04:50:53.529 INFO:journalctl@ceph.mon.a.smithi017.stdout:Mar 26 04:50:53 smithi017 bash[91250]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:50:53.660 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T04:50:53.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:50:53.744 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T04:50:53.744 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T04:50:53.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.c 2024-03-26T04:50:54.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:53 smithi017 systemd[1]: Stopping Ceph mon.c for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:50:54.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c[67138]: 2024-03-26T04:50:54.027+0000 7f768b801700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:50:54.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c[67138]: 2024-03-26T04:50:54.027+0000 7f768b801700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-26T04:50:54.434 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:54 smithi017 podman[91381]: 2024-03-26 04:50:54.171324326 +0000 UTC m=+0.242456266 container died 2c7ea6faebf3d19832fa3075a1d7847bd31007348ace89f1de6e0dba8e33bf06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T04:50:54.702 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:54 smithi017 podman[91381]: 2024-03-26 04:50:54.528270955 +0000 UTC m=+0.599402897 container cleanup 2c7ea6faebf3d19832fa3075a1d7847bd31007348ace89f1de6e0dba8e33bf06 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T04:50:54.702 INFO:journalctl@ceph.mon.c.smithi017.stdout:Mar 26 04:50:54 smithi017 bash[91381]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-c 2024-03-26T04:50:54.950 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T04:50:54.986 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:50:54.987 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T04:50:54.987 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T04:50:54.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b 2024-03-26T04:50:55.288 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:55 smithi129 systemd[1]: Stopping Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:50:55.642 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b[57890]: 2024-03-26T04:50:55.289+0000 7fb93dd0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:50:55.642 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b[57890]: 2024-03-26T04:50:55.289+0000 7fb93dd0d700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-26T04:50:55.642 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:55 smithi129 podman[75977]: 2024-03-26 04:50:55.52535462 +0000 UTC m=+0.338373686 container died fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:50:56.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:55 smithi129 podman[75977]: 2024-03-26 04:50:55.965380637 +0000 UTC m=+0.778399698 container cleanup fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T04:50:56.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:55 smithi129 bash[75977]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b 2024-03-26T04:50:56.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T04:50:56.400 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:56 smithi129 podman[75992]: 2024-03-26 04:50:56.150494919 +0000 UTC m=+0.618405332 container remove fe20e3dd00320f8d7ef32d0cc04a1c2c98f17634052f2e1acb136f30d7f6f1b2 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mon-b, RELEASE=squid-5e70124, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T04:50:56.400 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:56 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T04:50:56.400 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:56 smithi129 systemd[1]: Stopped Ceph mon.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:50:56.400 INFO:journalctl@ceph.mon.b.smithi129.stdout:Mar 26 04:50:56 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@mon.b.service: Consumed 5.294s CPU time. 2024-03-26T04:50:56.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:50:56.426 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T04:50:56.426 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T04:50:56.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:50:56.702 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:56 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y[60295]: ::ffff:172.21.15.129 - - [26/Mar/2024:04:50:56] "GET /metrics HTTP/1.1" 200 38700 "" "Prometheus/2.33.4" 2024-03-26T04:50:56.703 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:56 smithi017 systemd[1]: Stopping Ceph mgr.y for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:50:57.024 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:56 smithi017 podman[91501]: 2024-03-26 04:50:56.748801785 +0000 UTC m=+0.136005680 container died 4836c6142fed2f2f8ebfce0da45f9c0834c84beb8d012f821f4a49ae628ac017 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-03-26T04:50:57.332 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:57 smithi017 podman[91501]: 2024-03-26 04:50:57.101725961 +0000 UTC m=+0.488929852 container cleanup 4836c6142fed2f2f8ebfce0da45f9c0834c84beb8d012f821f4a49ae628ac017 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:50:57.333 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Mar 26 04:50:57 smithi017 bash[91501]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-y 2024-03-26T04:50:57.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T04:50:57.514 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:50:57.514 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T04:50:57.514 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T04:50:57.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x 2024-03-26T04:50:57.847 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:50:57 smithi129 systemd[1]: Stopping Ceph mgr.x for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:50:58.142 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:50:57 smithi129 podman[76098]: 2024-03-26 04:50:57.850848371 +0000 UTC m=+0.138907457 container died 7a621834387fe27744a663186124fae2b7aa0f06fef8187806c56c21e38c94c0 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124) 2024-03-26T04:50:58.459 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:50:58 smithi129 podman[76098]: 2024-03-26 04:50:58.228343915 +0000 UTC m=+0.516403002 container cleanup 7a621834387fe27744a663186124fae2b7aa0f06fef8187806c56c21e38c94c0 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T04:50:58.459 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Mar 26 04:50:58 smithi129 bash[76098]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:50:58.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T04:50:58.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:50:58.625 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T04:50:58.625 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T04:50:58.625 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.0 2024-03-26T04:50:58.919 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:50:58 smithi017 systemd[1]: Stopping Ceph osd.0 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:50:59.202 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:50:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:50:58.922+0000 7f74ac1db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:50:59.202 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:50:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:50:58.922+0000 7f74ac1db700 -1 osd.0 153 *** Got signal Terminated *** 2024-03-26T04:50:59.203 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:50:58 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0[70536]: 2024-03-26T04:50:58.922+0000 7f74ac1db700 -1 osd.0 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:04.646 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:04 smithi017 podman[91622]: 2024-03-26 04:51:04.386491079 +0000 UTC m=+5.560535162 container died f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True) 2024-03-26T04:51:04.938 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:04 smithi017 podman[91622]: 2024-03-26 04:51:04.722718852 +0000 UTC m=+5.896762943 container cleanup f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T04:51:04.938 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:04 smithi017 bash[91622]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0 2024-03-26T04:51:05.202 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:04 smithi017 podman[91639]: 2024-03-26 04:51:04.941025076 +0000 UTC m=+0.547912381 container remove f4afeb9d64a61bc3762d0908b61d453037fc3bfa8a7bafd5cba7a758282b49ca (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:05.658 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:05 smithi017 podman[91732]: 2024-03-26 04:51:05.246111669 +0000 UTC m=+0.017924502 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:05.658 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:05 smithi017 podman[91732]: 2024-03-26 04:51:05.350256217 +0000 UTC m=+0.122069081 container create 070260c3acc124dc728233644699356675917d915ac27049262e522049f6f0f9 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T04:51:05.658 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:05 smithi017 podman[91732]: 2024-03-26 04:51:05.58608761 +0000 UTC m=+0.357900433 container init 070260c3acc124dc728233644699356675917d915ac27049262e522049f6f0f9 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:51:05.658 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:05 smithi017 podman[91732]: 2024-03-26 04:51:05.589158498 +0000 UTC m=+0.360971326 container start 070260c3acc124dc728233644699356675917d915ac27049262e522049f6f0f9 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:51:05.658 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:05 smithi017 podman[91732]: 2024-03-26 04:51:05.660942083 +0000 UTC m=+0.432754916 container attach 070260c3acc124dc728233644699356675917d915ac27049262e522049f6f0f9 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:05.952 INFO:journalctl@ceph.osd.0.smithi017.stdout:Mar 26 04:51:05 smithi017 podman[91732]: 2024-03-26 04:51:05.88495987 +0000 UTC m=+0.656772693 container died 070260c3acc124dc728233644699356675917d915ac27049262e522049f6f0f9 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:51:06.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T04:51:06.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:06.492 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T04:51:06.492 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T04:51:06.492 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.1 2024-03-26T04:51:06.686 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:06 smithi017 systemd[1]: Stopping Ceph osd.1 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:06.952 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:51:06.834+0000 7fc43ec04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:06.952 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:51:06.834+0000 7fc43ec04700 -1 osd.1 153 *** Got signal Terminated *** 2024-03-26T04:51:06.953 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:06 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1[73017]: 2024-03-26T04:51:06.834+0000 7fc43ec04700 -1 osd.1 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:07.207 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:51:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:51:07.103Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-03-26T04:51:07.207 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:51:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:51:07.103Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-03-26T04:51:07.207 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:51:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:51:07.103Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-03-26T04:51:07.207 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:51:07 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:51:07.103Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-03-26T04:51:12.454 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:12 smithi017 podman[91838]: 2024-03-26 04:51:12.201607629 +0000 UTC m=+5.470671497 container died c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:12.740 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:12 smithi017 podman[91838]: 2024-03-26 04:51:12.534403953 +0000 UTC m=+5.803467815 container cleanup c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:12.740 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:12 smithi017 bash[91838]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1 2024-03-26T04:51:13.155 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:12 smithi017 podman[91867]: 2024-03-26 04:51:12.743991339 +0000 UTC m=+0.539013552 container remove c04a08104bf34bdb91c1c152d113029de129d588ec4b6439c985939cfbb67793 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T04:51:13.452 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:13 smithi017 podman[91959]: 2024-03-26 04:51:13.061241816 +0000 UTC m=+0.018070674 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:13.452 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:13 smithi017 podman[91959]: 2024-03-26 04:51:13.183103192 +0000 UTC m=+0.139932033 container create 41d57dc814da4464e232349377075597b35c21148902178f9a465d1a12bbb54a (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T04:51:13.452 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:13 smithi017 podman[91959]: 2024-03-26 04:51:13.380886089 +0000 UTC m=+0.337714934 container init 41d57dc814da4464e232349377075597b35c21148902178f9a465d1a12bbb54a (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T04:51:13.453 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:13 smithi017 podman[91959]: 2024-03-26 04:51:13.383967815 +0000 UTC m=+0.340796662 container start 41d57dc814da4464e232349377075597b35c21148902178f9a465d1a12bbb54a (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T04:51:13.952 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:13 smithi017 podman[91959]: 2024-03-26 04:51:13.481118911 +0000 UTC m=+0.437947775 container attach 41d57dc814da4464e232349377075597b35c21148902178f9a465d1a12bbb54a (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T04:51:13.952 INFO:journalctl@ceph.osd.1.smithi017.stdout:Mar 26 04:51:13 smithi017 podman[91959]: 2024-03-26 04:51:13.682921015 +0000 UTC m=+0.639749858 container died 41d57dc814da4464e232349377075597b35c21148902178f9a465d1a12bbb54a (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T04:51:14.323 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:14.240Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:14.323 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:14.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:14.346 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T04:51:14.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:14.390 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T04:51:14.390 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T04:51:14.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:51:14.692 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:14 smithi017 systemd[1]: Stopping Ceph osd.2 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:14.952 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:51:14.695+0000 7efe8e2d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:14.952 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:51:14.695+0000 7efe8e2d8700 -1 osd.2 153 *** Got signal Terminated *** 2024-03-26T04:51:14.952 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:14 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2[75489]: 2024-03-26T04:51:14.695+0000 7efe8e2d8700 -1 osd.2 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:20.295 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:20 smithi017 podman[92066]: 2024-03-26 04:51:20.007985685 +0000 UTC m=+5.392975486 container died 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T04:51:20.652 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:20 smithi017 podman[92066]: 2024-03-26 04:51:20.396148542 +0000 UTC m=+5.781138342 container cleanup 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:20.652 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:20 smithi017 bash[92066]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:51:20.952 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:20 smithi017 podman[92083]: 2024-03-26 04:51:20.656130211 +0000 UTC m=+0.641114391 container remove 53714135178c6a6518552f0ac5bc34ce545923bb7441c32aa4f24b3dd9526b05 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-03-26T04:51:21.314 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:21 smithi017 podman[92174]: 2024-03-26 04:51:20.910591868 +0000 UTC m=+0.018265436 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:21.386 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:21 smithi017 podman[92174]: 2024-03-26 04:51:21.048524912 +0000 UTC m=+0.156198479 container create c228080f75c318640ebcee1ef2bbdab13129230dfb1f3510fd65534118dc8dac (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T04:51:21.386 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:21 smithi017 podman[92174]: 2024-03-26 04:51:21.242551353 +0000 UTC m=+0.350224921 container init c228080f75c318640ebcee1ef2bbdab13129230dfb1f3510fd65534118dc8dac (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:51:21.386 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:21 smithi017 podman[92174]: 2024-03-26 04:51:21.245551462 +0000 UTC m=+0.353225030 container start c228080f75c318640ebcee1ef2bbdab13129230dfb1f3510fd65534118dc8dac (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T04:51:21.386 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:21 smithi017 podman[92174]: 2024-03-26 04:51:21.317605673 +0000 UTC m=+0.425279261 container attach c228080f75c318640ebcee1ef2bbdab13129230dfb1f3510fd65534118dc8dac (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.schema-version=1.0) 2024-03-26T04:51:21.702 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:21 smithi017 podman[92174]: 2024-03-26 04:51:21.545087607 +0000 UTC m=+0.652761175 container died c228080f75c318640ebcee1ef2bbdab13129230dfb1f3510fd65534118dc8dac (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T04:51:22.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T04:51:22.113 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:22 smithi017 podman[92200]: 2024-03-26 04:51:22.071092008 +0000 UTC m=+0.521070573 container remove c228080f75c318640ebcee1ef2bbdab13129230dfb1f3510fd65534118dc8dac (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T04:51:22.113 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:22 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-26T04:51:22.113 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:22 smithi017 systemd[1]: Stopped Ceph osd.2 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:51:22.113 INFO:journalctl@ceph.osd.2.smithi017.stdout:Mar 26 04:51:22 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.2.service: Consumed 5.258s CPU time. 2024-03-26T04:51:22.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:22.129 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T04:51:22.129 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T04:51:22.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3 2024-03-26T04:51:22.452 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:22 smithi017 systemd[1]: Stopping Ceph osd.3 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:22.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:22 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:51:22.457+0000 7f84acb54700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:22.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:22 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:51:22.457+0000 7f84acb54700 -1 osd.3 153 *** Got signal Terminated *** 2024-03-26T04:51:22.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:22 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:51:22.457+0000 7f84acb54700 -1 osd.3 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:24.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:24 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:51:24.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:24.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:24 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:24.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:24 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:25.392 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:25 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:25.102+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:26.395 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:26 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:26.139+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:26.396 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:26 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:26.212+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:27.452 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:27 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3[77962]: 2024-03-26T04:51:27.150+0000 7f84a9148700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.889136+0000 front 2024-03-26T04:51:02.889203+0000 (oldest deadline 2024-03-26T04:51:26.389191+0000) 2024-03-26T04:51:27.642 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:27.161+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:27.643 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:27 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:27.192+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:28.213 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:27 smithi017 podman[92281]: 2024-03-26 04:51:27.956774477 +0000 UTC m=+5.600864652 container died 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:51:28.571 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:28 smithi017 podman[92281]: 2024-03-26 04:51:28.307475431 +0000 UTC m=+5.951565613 container cleanup 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:51:28.571 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:28 smithi017 bash[92281]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3 2024-03-26T04:51:28.642 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:28.148+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:28.645 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:28.347+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:28.646 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:28 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:28.235+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:28.897 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:28 smithi017 podman[92298]: 2024-03-26 04:51:28.575123517 +0000 UTC m=+0.620688064 container remove 61ddd8e4617eda6d274c1b445b7d6daa06e615863fd9e84da2a83469939b5205 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:29.183 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:28 smithi017 podman[92388]: 2024-03-26 04:51:28.803812405 +0000 UTC m=+0.017499311 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:29.183 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:28 smithi017 podman[92388]: 2024-03-26 04:51:28.909441147 +0000 UTC m=+0.123128082 container create 158c5e9710e1fa99b5347bcc22a5ef8d29f9f62b8a73f6846e27b49c7fb72e9d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:29.183 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 podman[92388]: 2024-03-26 04:51:29.111704311 +0000 UTC m=+0.325391204 container init 158c5e9710e1fa99b5347bcc22a5ef8d29f9f62b8a73f6846e27b49c7fb72e9d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:51:29.183 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 podman[92388]: 2024-03-26 04:51:29.115465662 +0000 UTC m=+0.329152555 container start 158c5e9710e1fa99b5347bcc22a5ef8d29f9f62b8a73f6846e27b49c7fb72e9d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:29.183 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 podman[92388]: 2024-03-26 04:51:29.18664942 +0000 UTC m=+0.400336314 container attach 158c5e9710e1fa99b5347bcc22a5ef8d29f9f62b8a73f6846e27b49c7fb72e9d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T04:51:29.391 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:29.112+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:29.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:29.085+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:29.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:29.240+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:29.452 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 podman[92388]: 2024-03-26 04:51:29.425852204 +0000 UTC m=+0.639539097 container died 158c5e9710e1fa99b5347bcc22a5ef8d29f9f62b8a73f6846e27b49c7fb72e9d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T04:51:29.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:29 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:29.394+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:29.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T04:51:29.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 podman[92414]: 2024-03-26 04:51:29.906905658 +0000 UTC m=+0.474923279 container remove 158c5e9710e1fa99b5347bcc22a5ef8d29f9f62b8a73f6846e27b49c7fb72e9d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T04:51:29.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T04:51:29.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 systemd[1]: Stopped Ceph osd.3 for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:51:29.952 INFO:journalctl@ceph.osd.3.smithi017.stdout:Mar 26 04:51:29 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.3.service: Consumed 7.048s CPU time. 2024-03-26T04:51:29.970 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:29.971 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T04:51:29.971 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T04:51:29.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.4 2024-03-26T04:51:30.374 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:30.091+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:30.375 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:30 smithi129 systemd[1]: Stopping Ceph osd.4 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:30.375 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:30.123+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:30.375 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:30.244+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:30.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:30.377+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:30.642 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:30.496+0000 7f04fe39c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:30.642 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:30.496+0000 7f04fe39c700 -1 osd.4 153 *** Got signal Terminated *** 2024-03-26T04:51:30.642 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:30 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:30.496+0000 7f04fe39c700 -1 osd.4 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:31.392 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:31.116+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:31.393 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:31.220+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:31.393 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:31.101+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:31.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:31 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:31.421+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:32.392 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:32.082+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:32.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:32.125+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:32.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:32.189+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:32.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:32 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:32.463+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:33.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:33.134+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:33.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:33.239+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:33.392 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:33.083+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:33.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:33.461+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:33.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:33 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:33.461+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:34.391 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:34.179+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:34.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:34.231+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:34.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:34.231+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:34.392 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:34.086+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:34.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:51:34.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:34.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:34.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:34.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:34 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:34.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:34.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:34.423+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:34.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:34 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:34.423+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:35.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:35.133+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:35.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:35.133+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:35.392 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4[61174]: 2024-03-26T04:51:35.130+0000 7f04f6988700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:50:59.412792+0000 front 2024-03-26T04:50:59.412805+0000 (oldest deadline 2024-03-26T04:51:24.712647+0000) 2024-03-26T04:51:35.392 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:35.375+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:35.393 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:35.375+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:35.393 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:35.276+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:35.393 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:35 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:35.276+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:36.246 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:35 smithi129 podman[76233]: 2024-03-26 04:51:35.976607375 +0000 UTC m=+5.805317001 container died cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:36.247 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:36.145+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:36.247 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:36.145+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:36.619 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:36.373+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:36.620 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:36.373+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:36.620 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:36.258+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:36.620 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:36 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:36.258+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:36.620 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:36 smithi129 podman[76233]: 2024-03-26 04:51:36.413994761 +0000 UTC m=+6.242704395 container cleanup cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:51:36.620 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:36 smithi129 bash[76233]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4 2024-03-26T04:51:36.891 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:36 smithi129 podman[76250]: 2024-03-26 04:51:36.623642611 +0000 UTC m=+0.647753453 container remove cc81d25035e91d165726c9173fd2754fb3ec20ae1429343a5830296668c92ff4 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:51:37.306 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:37 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:37.109+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:37.306 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:37 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:37.109+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:37 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:37.224+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:37 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:37.224+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:37 smithi129 podman[76341]: 2024-03-26 04:51:37.024569331 +0000 UTC m=+0.113460629 container create 2b29f4daf23c274b18de805dd758c8d4a12ce9f75cba9a8437679aa67131f7ad (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:37 smithi129 podman[76341]: 2024-03-26 04:51:36.928640693 +0000 UTC m=+0.017532003 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:37 smithi129 podman[76341]: 2024-03-26 04:51:37.235333807 +0000 UTC m=+0.324225106 container init 2b29f4daf23c274b18de805dd758c8d4a12ce9f75cba9a8437679aa67131f7ad (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:37 smithi129 podman[76341]: 2024-03-26 04:51:37.238537952 +0000 UTC m=+0.327429258 container start 2b29f4daf23c274b18de805dd758c8d4a12ce9f75cba9a8437679aa67131f7ad (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T04:51:37.307 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:37 smithi129 podman[76341]: 2024-03-26 04:51:37.310179057 +0000 UTC m=+0.399070371 container attach 2b29f4daf23c274b18de805dd758c8d4a12ce9f75cba9a8437679aa67131f7ad (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T04:51:37.642 INFO:journalctl@ceph.osd.4.smithi129.stdout:Mar 26 04:51:37 smithi129 podman[76341]: 2024-03-26 04:51:37.527773022 +0000 UTC m=+0.616664330 container died 2b29f4daf23c274b18de805dd758c8d4a12ce9f75cba9a8437679aa67131f7ad (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319) 2024-03-26T04:51:37.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:37 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:37.325+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:37.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:37 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:37.325+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:38.110 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:38.060+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:38.110 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:38.060+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:38.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T04:51:38.170 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:38.171 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T04:51:38.171 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T04:51:38.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.5 2024-03-26T04:51:38.384 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:38.267+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:38.384 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:38.267+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:38.384 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:38 smithi129 systemd[1]: Stopping Ceph osd.5 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:38.384 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:38.312+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:38.385 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:38.312+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:38.641 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:38.533+0000 7fa530cb5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:38.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:38.533+0000 7fa530cb5700 -1 osd.5 153 *** Got signal Terminated *** 2024-03-26T04:51:38.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:38 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:38.533+0000 7fa530cb5700 -1 osd.5 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:39.303 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:39.027+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:39.303 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:39.027+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:39.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:39.309+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:39.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:39.309+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:39.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:39.339+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:39.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:39.339+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:40.324 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:39.984+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:40.324 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:39 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:39.984+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:40.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:40.327+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:40.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:40.327+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:40.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:40.338+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:40.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:40.338+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:40.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:40.338+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:41.326 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:40.971+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:41.326 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:40 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:40.971+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:41.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:41 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:41.329+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:41.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:41 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:41.329+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:41.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:41 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:41.367+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:41.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:41 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:41.367+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:41.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:41 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:41.367+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:42.291 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:42.011+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:42.291 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:42.011+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:42.291 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:42.011+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:42.641 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:42.293+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:42.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:42.294+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:42.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:42.395+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:42.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:42.395+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:42.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:42.395+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:43.391 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:43.280+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.529198+0000 front 2024-03-26T04:51:02.529182+0000 (oldest deadline 2024-03-26T04:51:27.828930+0000) 2024-03-26T04:51:43.392 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:43.280+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:07.829448+0000 front 2024-03-26T04:51:07.829426+0000 (oldest deadline 2024-03-26T04:51:32.529435+0000) 2024-03-26T04:51:43.392 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5[63519]: 2024-03-26T04:51:43.280+0000 7fa52d2a9700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:17.130820+0000 front 2024-03-26T04:51:17.130844+0000 (oldest deadline 2024-03-26T04:51:43.030992+0000) 2024-03-26T04:51:43.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:43.045+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:43.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:43.045+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:43.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:43.045+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:43.892 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:43.415+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:43.892 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:43.415+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:43.892 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:43 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:43.415+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:43.952 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:43.543Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:43.952 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:43 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:43.544Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:44.298 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:44 smithi129 podman[76448]: 2024-03-26 04:51:44.030403513 +0000 UTC m=+5.607212231 container died 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T04:51:44.298 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:44.031+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:44.298 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:44.031+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:44.298 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:44.031+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:44.641 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:44 smithi129 podman[76448]: 2024-03-26 04:51:44.507025961 +0000 UTC m=+6.083834687 container cleanup 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:44.642 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:44 smithi129 bash[76448]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5 2024-03-26T04:51:44.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:44.378+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:44.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:44.378+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:44.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:44 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:44.378+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:44.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:51:44.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:44.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:44.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:44.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:44 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:44.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:45.056 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:44 smithi129 podman[76478]: 2024-03-26 04:51:44.751773972 +0000 UTC m=+0.718348559 container remove 0e80c6fa26687ffd7230aa2d3e2a6cc332e562222f1bc2f30f1d4a6ee585759d (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:45.310 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:45 smithi129 podman[76575]: 2024-03-26 04:51:44.975554849 +0000 UTC m=+0.018021202 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:45.310 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:45 smithi129 podman[76575]: 2024-03-26 04:51:45.103061773 +0000 UTC m=+0.145528158 container create 916fa14ed9eae60f9c4e98a954ec54ed1cc4bb378fa599356b73bea7777af078 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:51:45.310 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:45.059+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:45.310 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:45.059+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:45.310 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:45.059+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:45.610 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:45 smithi129 podman[76575]: 2024-03-26 04:51:45.313517827 +0000 UTC m=+0.355984173 container init 916fa14ed9eae60f9c4e98a954ec54ed1cc4bb378fa599356b73bea7777af078 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T04:51:45.610 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:45 smithi129 podman[76575]: 2024-03-26 04:51:45.316762863 +0000 UTC m=+0.359229205 container start 916fa14ed9eae60f9c4e98a954ec54ed1cc4bb378fa599356b73bea7777af078 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:51:45.610 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:45 smithi129 podman[76575]: 2024-03-26 04:51:45.413538961 +0000 UTC m=+0.456005317 container attach 916fa14ed9eae60f9c4e98a954ec54ed1cc4bb378fa599356b73bea7777af078 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T04:51:45.610 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:45.339+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:45.610 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:45.339+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:45.610 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:45 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:45.339+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:45.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:45 smithi129 podman[76575]: 2024-03-26 04:51:45.614324074 +0000 UTC m=+0.656790424 container died 916fa14ed9eae60f9c4e98a954ec54ed1cc4bb378fa599356b73bea7777af078 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:51:46.156 INFO:journalctl@ceph.osd.5.smithi129.stdout:Mar 26 04:51:46 smithi129 podman[76600]: 2024-03-26 04:51:46.14186701 +0000 UTC m=+0.522325785 container remove 916fa14ed9eae60f9c4e98a954ec54ed1cc4bb378fa599356b73bea7777af078 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:46.156 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:46.037+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:46.156 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:46.037+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:46.156 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:46.037+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:46.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T04:51:46.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:46.204 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T04:51:46.204 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T04:51:46.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.6 2024-03-26T04:51:46.516 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 systemd[1]: Stopping Ceph osd.6 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:46.516 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:46.321+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:46.516 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:46.321+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:46.516 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:46.321+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:46.892 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:46.519+0000 7f67b0270700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:46.892 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:46.519+0000 7f67b0270700 -1 osd.6 153 *** Got signal Terminated *** 2024-03-26T04:51:46.892 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:46 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:46.519+0000 7f67b0270700 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:47.365 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:47.009+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:47.365 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:47.009+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:47.365 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:47.009+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:47.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:47.368+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:47.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:47.368+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:47.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:47.368+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:48.341 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:47.968+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:48.342 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:47.968+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:48.342 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:47 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:47.968+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:48.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:48.344+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:48.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:48.344+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:48.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:48.344+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:49.142 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:48.963+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:49.142 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:48.963+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:49.142 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:48.963+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:49.142 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:48 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:48.963+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.030109+0000 front 2024-03-26T04:51:27.030172+0000 (oldest deadline 2024-03-26T04:51:48.129929+0000) 2024-03-26T04:51:49.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:49.330+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:49.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:49.330+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:49.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:49.330+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:50.324 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:49.954+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:50.324 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:49.954+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:50.324 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:49.954+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:50.324 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:49 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:49.954+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.030109+0000 front 2024-03-26T04:51:27.030172+0000 (oldest deadline 2024-03-26T04:51:48.129929+0000) 2024-03-26T04:51:50.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:50.327+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:50.643 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:50.327+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:50.643 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:50.327+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:51.316 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:50.936+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.527290+0000 front 2024-03-26T04:51:02.527420+0000 (oldest deadline 2024-03-26T04:51:28.427057+0000) 2024-03-26T04:51:51.316 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:50.936+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:08.928282+0000 front 2024-03-26T04:51:08.928350+0000 (oldest deadline 2024-03-26T04:51:34.228111+0000) 2024-03-26T04:51:51.316 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:50.936+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:18.329072+0000 front 2024-03-26T04:51:18.329088+0000 (oldest deadline 2024-03-26T04:51:41.228907+0000) 2024-03-26T04:51:51.316 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:50 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6[65848]: 2024-03-26T04:51:50.936+0000 7f67a885c700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.030109+0000 front 2024-03-26T04:51:27.030172+0000 (oldest deadline 2024-03-26T04:51:48.129929+0000) 2024-03-26T04:51:51.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:51.319+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:51.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:51.319+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:51.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:51.319+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:51.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:51 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:51.319+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:52.142 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:51 smithi129 podman[76679]: 2024-03-26 04:51:51.848608435 +0000 UTC m=+5.421896106 container died 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:52.335+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:52.335+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:52.335+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:52 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:52.335+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 podman[76679]: 2024-03-26 04:51:52.227970582 +0000 UTC m=+5.801258243 container cleanup 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 bash[76679]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6 2024-03-26T04:51:52.642 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 podman[76696]: 2024-03-26 04:51:52.47089089 +0000 UTC m=+0.621745151 container remove 8d9b6172909c83ee0de16ac1e8c6c885769df1a8c9e2e3366fcd3f672fb20c0e (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T04:51:53.062 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 podman[76788]: 2024-03-26 04:51:52.796786188 +0000 UTC m=+0.112088511 container create bf8530aeee0ff65a6a0079fea0e6fd94cd94e3568bc7c1dd52de36354b8e0fed (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T04:51:53.062 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 podman[76788]: 2024-03-26 04:51:52.70287452 +0000 UTC m=+0.018176874 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:51:53.062 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 podman[76788]: 2024-03-26 04:51:52.990822247 +0000 UTC m=+0.306124570 container init bf8530aeee0ff65a6a0079fea0e6fd94cd94e3568bc7c1dd52de36354b8e0fed (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T04:51:53.062 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:52 smithi129 podman[76788]: 2024-03-26 04:51:52.993772972 +0000 UTC m=+0.309075297 container start bf8530aeee0ff65a6a0079fea0e6fd94cd94e3568bc7c1dd52de36354b8e0fed (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:51:53.062 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:53 smithi129 podman[76788]: 2024-03-26 04:51:53.065564877 +0000 UTC m=+0.380867201 container attach bf8530aeee0ff65a6a0079fea0e6fd94cd94e3568bc7c1dd52de36354b8e0fed (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:51:53.392 INFO:journalctl@ceph.osd.6.smithi129.stdout:Mar 26 04:51:53 smithi129 podman[76788]: 2024-03-26 04:51:53.291502402 +0000 UTC m=+0.606804729 container died bf8530aeee0ff65a6a0079fea0e6fd94cd94e3568bc7c1dd52de36354b8e0fed (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T04:51:53.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:53.299+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:53.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:53.299+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:53.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:53.299+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:53.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:53 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:53.299+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:53.952 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:51:53.544Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:53.952 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:53.544Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:53.953 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:53 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:53.545Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:54.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T04:51:54.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:51:54.050 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T04:51:54.050 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T04:51:54.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.7 2024-03-26T04:51:54.291 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 systemd[1]: Stopping Ceph osd.7 for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:51:54.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.293+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:54.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.293+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:54.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.293+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:54.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.293+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:54.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.397+0000 7f418f562700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:51:54.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.397+0000 7f418f562700 -1 osd.7 153 *** Got signal Terminated *** 2024-03-26T04:51:54.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:54 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:54.397+0000 7f418f562700 -1 osd.7 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:51:54.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:51:54.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:54.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:51:54.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:51:54 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:51:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:51:55.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:55.284+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:55.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:55.284+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:55.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:55.284+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:55.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:55 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:55.284+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:56.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:56.286+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:56.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:56.286+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:56.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:56.286+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:56.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:56 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:56.286+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:57.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:57.247+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:57.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:57.247+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:57.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:57.247+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:57.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:57 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:57.247+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:58.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:58.274+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:58.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:58.274+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:58.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:58.274+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:58.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:58 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:58.274+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:51:59.641 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:59.318+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-03-26T04:51:02.063281+0000 front 2024-03-26T04:51:02.063192+0000 (oldest deadline 2024-03-26T04:51:26.163130+0000) 2024-03-26T04:51:59.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:59.318+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-03-26T04:51:09.664251+0000 front 2024-03-26T04:51:09.664179+0000 (oldest deadline 2024-03-26T04:51:33.764033+0000) 2024-03-26T04:51:59.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:59.318+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-03-26T04:51:19.065132+0000 front 2024-03-26T04:51:19.065071+0000 (oldest deadline 2024-03-26T04:51:40.164903+0000) 2024-03-26T04:51:59.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:59 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7[68178]: 2024-03-26T04:51:59.318+0000 7f418bb56700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-03-26T04:51:27.166317+0000 front 2024-03-26T04:51:27.166435+0000 (oldest deadline 2024-03-26T04:51:50.666277+0000) 2024-03-26T04:52:00.094 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:51:59 smithi129 podman[76893]: 2024-03-26 04:51:59.823329806 +0000 UTC m=+5.548263451 container died f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T04:52:00.391 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:00 smithi129 podman[76893]: 2024-03-26 04:52:00.272308104 +0000 UTC m=+5.997241799 container cleanup f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:52:00.392 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:00 smithi129 bash[76893]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7 2024-03-26T04:52:00.881 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:00 smithi129 podman[76911]: 2024-03-26 04:52:00.565291115 +0000 UTC m=+0.741846551 container remove f505e855258d1a12b8d1bab4cb12c3d75a2ec106fcdaf0d5bcd418e9d6e7ba58 (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T04:52:01.141 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:00 smithi129 podman[77003]: 2024-03-26 04:52:00.788090339 +0000 UTC m=+0.017816547 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:52:01.142 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:00 smithi129 podman[77003]: 2024-03-26 04:52:00.907796965 +0000 UTC m=+0.137523144 container create 79cc960fea64af33a37f4b351abb6e3017516cc1b1f11d0f9ff664deaf1518af (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T04:52:01.142 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:01 smithi129 podman[77003]: 2024-03-26 04:52:01.11035867 +0000 UTC m=+0.340084856 container init 79cc960fea64af33a37f4b351abb6e3017516cc1b1f11d0f9ff664deaf1518af (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-26T04:52:01.142 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:01 smithi129 podman[77003]: 2024-03-26 04:52:01.113323107 +0000 UTC m=+0.343049288 container start 79cc960fea64af33a37f4b351abb6e3017516cc1b1f11d0f9ff664deaf1518af (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T04:52:01.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:01 smithi129 podman[77003]: 2024-03-26 04:52:01.193712353 +0000 UTC m=+0.423438549 container attach 79cc960fea64af33a37f4b351abb6e3017516cc1b1f11d0f9ff664deaf1518af (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:52:01.642 INFO:journalctl@ceph.osd.7.smithi129.stdout:Mar 26 04:52:01 smithi129 podman[77003]: 2024-03-26 04:52:01.411518565 +0000 UTC m=+0.641244749 container died 79cc960fea64af33a37f4b351abb6e3017516cc1b1f11d0f9ff664deaf1518af (image=quay.ceph.io/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-26T04:52:01.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T04:52:02.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:52:02.017 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T04:52:02.017 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T04:52:02.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T04:52:02.202 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 systemd[1]: Stopping Ceph prometheus.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T04:52:02.462 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:52:02.463 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T04:52:02.463 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:52:02.463 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:52:02.463 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T04:52:02.463 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a[53771]: ts=2024-03-26T04:52:02.206Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T04:52:02.463 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 podman[77082]: 2024-03-26 04:52:02.216536542 +0000 UTC m=+0.103195249 container died 807223168a4309fa7220a2802c457b599564bc5f1cfe859f2ac0e1ea6566b1e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:02.811 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 podman[77082]: 2024-03-26 04:52:02.597840293 +0000 UTC m=+0.484498996 container cleanup 807223168a4309fa7220a2802c457b599564bc5f1cfe859f2ac0e1ea6566b1e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:02.811 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 bash[77082]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:52:02.811 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Mar 26 04:52:02 smithi129 bash[77109]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T04:52:02.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-67a15696-eb29-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T04:52:02.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:52:02.855 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T04:52:02.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67a15696-eb29-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:52:03.661 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:52:03.545Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:52:03.661 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:52:03.545Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:52:03.661 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:03 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:52:03.546Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:52:04.363 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:04 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=error ts=2024-03-26T04:52:04.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:52:04.363 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:04 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:52:04.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-03-26T04:52:04.363 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:04 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=warn ts=2024-03-26T04:52:04.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi129.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.129:8443: connect: connection refused" 2024-03-26T04:52:04.702 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:04 smithi017 systemd[1]: Stopping Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:52:04.702 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:04 smithi017 podman[92753]: 2024-03-26 04:52:04.485380378 +0000 UTC m=+0.108515832 container died d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:05.097 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:04 smithi017 podman[92753]: 2024-03-26 04:52:04.824703582 +0000 UTC m=+0.447839033 container cleanup d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:05.097 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:04 smithi017 bash[92753]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T04:52:05.097 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:04 smithi017 bash[92778]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter.a" found: no such container 2024-03-26T04:52:05.399 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:05 smithi017 podman[92767]: 2024-03-26 04:52:05.10093527 +0000 UTC m=+0.619896642 container remove d9bfcddb743324beaae5cdb3aa78c7a5abeb6d732a63e28bb179965c50d8e59a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:05.400 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:05 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:52:05.400 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:05 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T04:52:05.400 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:05 smithi017 systemd[1]: Stopped Ceph node-exporter.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:52:05.400 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Mar 26 04:52:05 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 3.825s CPU time. 2024-03-26T04:52:05.400 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:05 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:52:05.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:05 smithi017 ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a[59046]: level=info ts=2024-03-26T04:52:05.403Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:52:05.702 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:05 smithi017 podman[92833]: 2024-03-26 04:52:05.455172887 +0000 UTC m=+0.147744821 container died 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:06.043 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:05 smithi017 podman[92833]: 2024-03-26 04:52:05.779312932 +0000 UTC m=+0.471884873 container cleanup 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:06.043 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:05 smithi017 bash[92833]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:52:06.043 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:05 smithi017 podman[92847]: 2024-03-26 04:52:05.963690022 +0000 UTC m=+0.513730241 container remove 0c94534064e5c83f10f21637ba2b02a997b6f7b3579fc7489e801575cadf862f (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:52:06.043 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:06 smithi017 podman[92847]: 2024-03-26 04:52:06.047659789 +0000 UTC m=+0.597700008 volume remove 34412dce14c8f7bcd47243475f46f4c6e73cd3020f8abed952fe413111e0bdfe 2024-03-26T04:52:06.340 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:06 smithi017 bash[92858]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T04:52:06.340 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:06 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:52:06.340 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:06 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:52:06.340 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Mar 26 04:52:06 smithi017 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.037s CPU time. 2024-03-26T04:52:40.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67a15696-eb29-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:52:41.891 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:41 smithi129 systemd[1]: Stopping Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:52:41.892 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:41 smithi129 podman[77441]: 2024-03-26 04:52:41.70116301 +0000 UTC m=+0.112403931 container died 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:52:42.303 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 podman[77441]: 2024-03-26 04:52:42.10711125 +0000 UTC m=+0.518352173 container cleanup 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:52:42.303 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 bash[77441]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T04:52:42.303 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 bash[77468]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter.b" found: no such container 2024-03-26T04:52:42.590 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:42 smithi129 systemd[1]: Stopping Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715... 2024-03-26T04:52:42.591 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 podman[77455]: 2024-03-26 04:52:42.308259326 +0000 UTC m=+0.610963090 container remove 1f9a7c9aca8fb2cd72a2b5fddfca053742d9debd8b95535cbe954157329824c9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:52:42.591 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:52:42.591 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T04:52:42.591 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 systemd[1]: Stopped Ceph node-exporter.b for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:52:42.591 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Mar 26 04:52:42 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 3.762s CPU time. 2024-03-26T04:52:42.866 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:42 smithi129 ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a[50995]: t=2024-03-26T04:52:42+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-26T04:52:42.867 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:42 smithi129 podman[77522]: 2024-03-26 04:52:42.60510937 +0000 UTC m=+0.089994192 container died 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.openshift.expose-services=, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8) 2024-03-26T04:52:43.141 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:42 smithi129 podman[77522]: 2024-03-26 04:52:42.961026451 +0000 UTC m=+0.445911283 container cleanup 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container) 2024-03-26T04:52:43.141 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:42 smithi129 bash[77522]: ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a 2024-03-26T04:52:43.141 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:42 smithi129 bash[77558]: Error: no container with name or ID "ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana.a" found: no such container 2024-03-26T04:52:43.641 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:43 smithi129 podman[77536]: 2024-03-26 04:52:43.281624766 +0000 UTC m=+0.678108345 container remove 45c2baa57c4d0b1b9c6d921194bbb7c4fa6459057a9251c7bb9ded5888aaff3e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-67a15696-eb29-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, version=8.5, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, distribution-scope=public, io.openshift.expose-services=, name=ubi8, vcs-type=git, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, architecture=x86_64) 2024-03-26T04:52:43.641 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:43 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T04:52:43.641 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:43 smithi129 systemd[1]: Stopped Ceph grafana.a for 67a15696-eb29-11ee-95d0-87774f69a715. 2024-03-26T04:52:43.641 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Mar 26 04:52:43 smithi129 systemd[1]: ceph-67a15696-eb29-11ee-95d0-87774f69a715@grafana.a.service: Consumed 4.132s CPU time. 2024-03-26T04:53:06.317 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:53:06.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:53:06.383 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T04:53:06.385 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/remote/smithi017/crash 2024-03-26T04:53:06.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:53:06.439 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:53:06.439 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:53:06.441 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/remote/smithi129/crash 2024-03-26T04:53:06.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:53:06.482 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/67a15696-eb29-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:53:06.482 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:53:06.484 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T04:53:06.484 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:53:06.515 INFO:teuthology.orchestra.run.smithi017.stdout:2024-03-26T04:31:32.298227+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:53:06.515 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T04:53:06.515 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[SEC\]' /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:53:06.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]' /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:53:06.657 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[WRN\]' /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-26T04:53:06.726 INFO:teuthology.orchestra.run.smithi017.stdout:2024-03-26T04:31:32.298227+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T04:53:06.727 INFO:tasks.cephadm:Compressing logs... 2024-03-26T04:53:06.728 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:53:06.770 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:53:06.795 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:: No such file or directory 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.smpl.smithi017.jmwydb.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.smpl.smithi017.jmwydb.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.audit.log: 76.8% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.smpl.smithi017.jmwydb.log.gz 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi017.yqadkl.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T04:53:06.796 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi017.yqadkl.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T04:53:06.797 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T04:53:06.797 INFO:teuthology.orchestra.run.smithi017.stderr: 76.5% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi017.yqadkl.log.gz 2024-03-26T04:53:06.798 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T04:53:06.798 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mgr.y.log: /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/tcmu-runner.log: 84.5% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T04:53:06.798 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:53:06.803 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:: No such file or directory 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi129.pgirvy.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi129.pgirvy.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr: 76.5% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi129.pgirvy.log.gz 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:53:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:53:06.805 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log: 90.5% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:53:06.805 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T04:53:06.805 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.0.log: 94.2% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:53:06.806 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-volume.log: 87.7% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:53:06.807 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.smpl.smithi129.jvfyul.log 2024-03-26T04:53:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.b.log: 88.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:53:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.smpl.smithi129.jvfyul.log: 76.7% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-client.rgw.smpl.smithi129.jvfyul.log.gz 2024-03-26T04:53:06.809 INFO:teuthology.orchestra.run.smithi017.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:53:06.809 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:53:06.809 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T04:53:06.809 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log: 90.6% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:53:06.810 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:53:06.812 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.2.log: 93.0% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:53:06.813 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T04:53:06.817 INFO:teuthology.orchestra.run.smithi129.stderr: 91.4% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:53:06.839 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:53:06.841 INFO:teuthology.orchestra.run.smithi129.stderr: 89.0% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T04:53:07.124 INFO:teuthology.orchestra.run.smithi129.stderr: 92.4% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T04:53:07.153 INFO:teuthology.orchestra.run.smithi017.stderr: 88.6% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T04:53:07.298 INFO:teuthology.orchestra.run.smithi017.stderr: 92.1% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T04:53:07.950 INFO:teuthology.orchestra.run.smithi129.stderr: 93.4% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T04:53:08.020 INFO:teuthology.orchestra.run.smithi017.stderr: 90.9% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T04:53:08.021 INFO:teuthology.orchestra.run.smithi017.stderr: 93.5% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T04:53:08.150 INFO:teuthology.orchestra.run.smithi129.stderr: 93.6% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T04:53:08.219 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T04:53:08.522 INFO:teuthology.orchestra.run.smithi017.stderr: 93.7% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T04:53:08.522 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T04:53:08.524 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-26T04:53:08.525 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m1.739s 2024-03-26T04:53:08.525 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m5.798s 2024-03-26T04:53:08.525 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m0.261s 2024-03-26T04:53:08.526 INFO:teuthology.orchestra.run.smithi017.stderr: 93.7% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T04:53:08.901 INFO:teuthology.orchestra.run.smithi017.stderr: 93.7% -- replaced with /var/log/ceph/67a15696-eb29-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T04:53:08.902 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-03-26T04:53:08.903 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m2.121s 2024-03-26T04:53:08.903 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m8.574s 2024-03-26T04:53:08.903 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.295s 2024-03-26T04:53:08.903 INFO:tasks.cephadm:Archiving logs... 2024-03-26T04:53:08.903 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/remote/smithi017/log 2024-03-26T04:53:08.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:53:09.953 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/remote/smithi129/log 2024-03-26T04:53:09.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:53:10.816 INFO:tasks.cephadm:Removing cluster... 2024-03-26T04:53:10.816 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67a15696-eb29-11ee-95d0-87774f69a715 --force 2024-03-26T04:53:11.110 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 67a15696-eb29-11ee-95d0-87774f69a715 --force 2024-03-26T04:53:11.400 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T04:53:11.400 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:53:11.415 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:53:11.429 INFO:tasks.cephadm:Teardown complete 2024-03-26T04:53:11.429 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T04:53:11.440 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T04:53:11.440 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:53:11.457 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:53:11.468 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:53:11.473 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:53:11.473 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-03-26T04:53:11.473 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +636us[ +629us] +/- 93ms 2024-03-26T04:53:11.473 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -592us[ -599us] +/- 60ms 2024-03-26T04:53:11.473 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +202us[ +202us] +/- 63ms 2024-03-26T04:53:11.473 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:53:11.483 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:53:11.488 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:53:11.488 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-26T04:53:11.488 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +615us[ +615us] +/- 93ms 2024-03-26T04:53:11.488 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -593us[ -576us] +/- 60ms 2024-03-26T04:53:11.488 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 5 +237us[ +237us] +/- 63ms 2024-03-26T04:53:11.488 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:53:11.488 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T04:53:11.497 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T04:53:11.498 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T04:53:11.527 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:53:11.610 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:53:11.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T04:53:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:53:11.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T04:53:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:53:11.748 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T04:53:11.758 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T04:53:11.785 INFO:teuthology.task.internal:Duration was 1692.031469 seconds 2024-03-26T04:53:11.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T04:53:11.818 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T04:53:11.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:53:11.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:53:11.861 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:53:11.864 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:53:12.239 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T04:53:12.239 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-03-26T04:53:12.239 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:53:12.299 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-26T04:53:12.299 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:53:12.327 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T04:53:12.328 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:53:12.341 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:53:12.401 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T04:53:12.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:53:12.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:53:12.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T04:53:12.842 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T04:53:12.842 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:53:12.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:53:12.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T04:53:12.904 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:53:12.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:53:12.933 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-03-26T04:53:12.959 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-26T04:53:13.011 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:53:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:53:13.025 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:53:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:53:13.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T04:53:13.074 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T04:53:13.075 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/remote/smithi017 2024-03-26T04:53:13.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:53:13.121 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018/remote/smithi129 2024-03-26T04:53:13.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:53:13.162 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T04:53:13.162 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:53:13.165 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:53:13.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T04:53:13.222 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T04:53:13.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T04:53:13.250 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T04:53:13.250 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:53:13.252 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:53:13.264 INFO:teuthology.orchestra.run.smithi017.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:53 /home/ubuntu/cephtest 2024-03-26T04:53:13.266 INFO:teuthology.orchestra.run.smithi129.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:53 /home/ubuntu/cephtest 2024-03-26T04:53:13.267 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T04:53:13.286 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T04:53:13.353 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T04:53:13.383 DEBUG:teuthology.nuke:shortname: smithi017 2024-03-26T04:53:13.384 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:53:13.408 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-26T04:53:13.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:53:13.415 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:15:24.923824', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjFUBGFRzeE04nKhP24Q/b3tXFhiqf2FLgvyVoweemiWgsFOoQms9A3hmEg7rWrT2goj041SVDVqqvOZrcvL30='} 2024-03-26T04:53:13.430 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7622018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 04:15:24.926714', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJAB26fsuA+6G1K/qO2UKQKU4QraxvCjg7fih7yVeO79+yZUDK+RGOjcpzw2CTHwULaS6ZOPCB2yLGygKsewIu8='} 2024-03-26T04:53:13.464 INFO:teuthology.orchestra.console.smithi017:Power off 2024-03-26T04:53:13.465 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:53:13.478 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-26T04:53:13.479 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:53:13.487 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2024-03-26T04:53:13.488 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:53:13.502 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-26T04:53:13.503 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:53:13.516 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-03-26T04:53:13.537 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-26T04:53:17.518 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:53:17.538 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:53:17.628 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-03-26T04:53:17.647 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-26T04:53:21.633 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:53:21.648 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:53:21.745 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2024-03-26T04:53:21.757 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-26T04:53:21.847 INFO:teuthology.orchestra.console.smithi017:Power off completed 2024-03-26T04:53:21.858 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-26T04:53:21.963 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 1692.0314693450928 failure_reason: '"2024-03-26T04:31:32.298227+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-26T04:53:21.963 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:53:22.045 INFO:teuthology.run:FAIL